Approved For Release ^(|^^|O^:^^-^^^-OO789ROO2£0Od9^Oi0te5t
To see whether the mean z score might be related to the number of experiments each investigator
ran, we performed a correlation between N and MEAN Z
(in Table
3).
Results were as follows:
Correlation
r-squared
t(2 1)
p
-0.0185
0.0003
-0.0941
0.9257
In summary, taking the data taken at face value
(i.e. not weighted by quality analysis), we can
make two
statements:
First,
considering
all
available
data,
there do
appear
to be
significant
differences among mean
z
scores
obtained by
different
experimenters.
Second,
there
is
a
nonsignificant correlation between the number of experiments run by principal investigators and
their mean
z scores. So to return to the question
at the beginning of this
section: Do different
experimenters obtain about the same results?
The answer
is no — experimenters in this survey
showed mean z scores ranging from -2.38 to 2.95.
As to the question of whether only one or two
individuals may be responsible
for the
overall
significance,
the answer
is
also
no\ 25%
of the
experimenters in our survey reported mean z scores beyond 2 and -2.
5. RNGs were nonrandom
This
criticism may be addressed by examining the
results of control
studies reported
in the
literature.
The
results shown
in Table
4 were compiled from
14 of the 44
detailed references
referred to in Table
3, and were contributed by the following twelve authors: Dunne (Princeton),
57 control studies; Schmidt,
23; Broughten,
8; Braud,
2; and one each for Bierman,
Hill, May,
Millar,
Morris,
Schechter, Honorton, and Palmer.
The other references did not report control
results in detail and could not be used.
Table
4:
Combined control studies
Data
Number of
control studies
X z’s
z
overall
z
p
(2-tail)
sd
Survey
-0.012
-0.0003
-0.002
0.999
1.036
Princeton
2.829
0.0496
0.375
0.708
.806
Combined
2.817
0.0287
0.285
0.776
.905
A variance test of the observed standard deviation (sd = .905) against the expected variance of
1.0 for 98 samples results in a chi-square = 80.2645 (97 df), z = -1.22, and p < .222 (2-tailed).
Thus,
for the references where control runs were described in sufficient detail to determine the
B-19
Approved For Release 20
'6-00789R002200390001 -5
EftrRelease
-00789R002200390001 -5
number of binary hits and trials, there
is no evidence of systematic (mean or variance) bias in the
RNG equipment.
QUALITY ANALYSIS: A PROPOSAL
In
this section, we address how we plan to judge the quality of the published experiments.
Quality analysis in effect adds a weighting factor to each experiment’s reported
z,
t, or p value,
depending on the
assessed
quality
of
that experiment.
To avoid making
a
subjective
quality
assessment
for each experiment,
criteria and
associated weights can be defined such
that
if a
criterion
is met, the weight associated with
that criterion
is added to that experiment’s
overall
weighting
factor.
Rosenthal
(1984, p.46-48)
describes
a
variety
of factors one might want to
consider when performing
quality
analyses, but
it
is
clear that the choice
of weighting
criteria
depends on the research context.
For the present analysis, Table 5 shows our initial proposal for
criteria and associated weights; these are explained following the table.
Table
5: Weighting criteria for RNG quality analysis
Criteria*
Weighting factors
Controls
With data
Without data
local control runs
global control runs
other control/random tests
target bit oscillation
Data Integrity
automatic
hit/trial counters
tamper resistant equipment
automatic data recording
Statistical Integrity
pre-specified analysis
fixed run lengths
direction of effort stated
Subject type
ordinary subjects
special subjects
experimenter as subject
Reporting clarity
fully reported hits or trials and z
report of
z, p, or t only
report of other statistics
“significant" only
“nonsignificant” only
* See text for explanation of criteria.
Explanation of RNG weighting criteria
Controls
In Table 5, a “local” control means the equipment was checked for randomness as
part of the experimental protocol. A typical design
is to have an experimental run followed by a
B-20
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
Meta-Analvsis Pan
Approved For Release yN«mS5!nCBP-00789R00220039b001 -5
control run equivalent
in
all respects to the experiment run, but where the subject
applies no
“effort” to the task or
is absent. A “global” control means the equipment (RNG, computer, etc.)
was
tested
under
the
same
conditions
as
used
in
the
experiment,
but
separate
from
the
experimental sessions.
“Other” control or randomness tests meant that some reference was made
to control runs or randomness tests, but the detailed results were either (a) not in the report or (b)
the explanation of the controls were referenced or related to a description in another article.
The
columns labeled “With data” and “Without data” show different weights assigned to control runs
depending on whether actual data were reported. “Target bit oscillation” means the assigned “hit”
bit alternated with each newly generated bit to counterbalance any possible RNG bias.
Data Integrity
The “automatic hit/trial counters” criterion
is satisfied
if the RNG equipment
has an automated method of keeping track of hits and trials.
"Automatic data recording" requires
use of punched paper tape, magnetic tape, computer disk, or so on, to automatically record the
data collected in the experiment.
There are instances in the literature
(especially in reports from
the early
1 970’s) where the automatic counter criterion
is met, but not automatic data recording.
“Tamper resistant equipment”
requires either that the RNG was
(a)
in a locked laboratory and
inaccessible to subjects at any time,
(b) the experiment was under the immediate supervision of an
experimenter,
(c) the equipment had a “fail-safe" or interlock system that prevented disruption of
or tampering with the data collection process, or (d) the device was a computer with software data
protection such as a password, protected
files, or so on.
Statistical Integrity
“Pre-specified
analysis” means
it
is
clear from the
report that the
statistical analysis method was defined before data was collected.
“Fixed run lengths" means the
total number of
trials was specified in advance of data
collection.
“Direction of effort stated"
requires that
it was clear whether the planned test was one-tailed or two-tailed, and what direction
of "effort” subjects were to aim for during the experiment.
Subject Integrity
This category checked whether the subjects used in the experiment were
ordinary,
selected or special
in some other way,
or the experimenter (s).
Stronger weight was
applied to unselected subjects because
it was felt they would have less invested in the experimental
outcome and would be less likely to intentionally or unintentionally interfere with the equipment or
procedures.
Reporting Integrity
If the report included the actual number of trials and hits, or the number
of trials and a
z, p, or t score, this was assigned the greatest weight.
If it included only
z, p, or t
scores, this was assigned less weight.
Report of any other statistics that we had to transform into the
equivalent of z scores were assigned the lowest weight.
In addition, reports consisting only of the
statement
"significant,”
without supporting data, were assigned a weight of 2 and
similarly, the
statement "nonsignificant” was assigned a weight of 4.
Method, of calculating quality-weighted analysis
The weighting factor per experiment would be calculated as follows:
If the criterion was clearly
present in the published report, the associated weighting factor would be added to that experiment’s
B-21
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001-5
^ApproveFfor Release
f^0L°A
:S5
Dl'PFE®°0789R002200390001 '5
weight.
If the criterion was not met, the weight assigned for that factor would be zero (0)
. The sum
of the individual weights would be the overall weight per experiment, and the final overall weighted
z score
is then calculated as
Weighted Z = Z
Wj
Zj
s/Zwi 2
(5)
Thus the minimum weight per experiment would be 0 if there were no mention of control tests,
no description indicating that data collection was protected in some way, no evidence that statistical
tests were pre-planned, insufficient report on who the subjects were, and no report of results. The
maximum weight would be 125 (sum of three control weights, three data integrity weights, three
statistical integrity weights, use of ordinary subjects, and
full report of data).
Weighting the filedrawer estimate
We propose to weight our estimated 95 nonsignificant filedrawer studies with the average weight
found in the rest of the studies.
This proposal has a potential criticism, however.
Our means of
estimating the filedrawer size depends on the observed z score distribution.
Since the individual z
scores depend on the weighting factors (which were in effect all l’s in the analysis reported in this
paper),
the unweighted
filedrawer
estimate may be
smaller than
a
similar estimate made with
weighted z scores, thus inflating the final results.
In response to this criticism, we would point out
that
the
quality
analysis
is
actually
orthogonal
to
the
filedrawer
estimate because
the
actual
magnitude of a z score does not change with our quality analysis, instead the importance of the z
score
is affected, and the importance of a z score
is not considered in our filedrawer estimation
method, only in the
final estimate of overall significance.
In addition, by adding a group of nonsignificant studies (the filedrawer estimate by definition is
composed
of nonsignificant
studies)
into
a pool
of
z
scores
that have already been weighted
according to quality, we are in effect creating an u/jra-conservative test. A case could be made, for
instance, on why a filedrawer estimate should not be added into a quality-weighted analysis at all,
but to take the conservative approach given the nature of the claim, we will pool the 95 estimated
studies along with the quality-weighted z scores.
Defining experiments in the Quality Analysis
Although adequate
for a
first-pass
analysis,
the method
of selecting experiments
described
above would be less than perfect for a quality-weighted analysis. The main objection that could be
raised is that the decision on what constitutes the subjects’ “direction of effort”
is dependent on the
reviewer’s
interpretation
of the experimental procedure.
In many
articles, we took educated
guesses to decide what were the
actual
conditions, what were the
subjects’
intentions,
did the
authors in fact predict in advance the outcome, and so on.
B-22
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00789R002200390001 -5