Approved For Release 209 NPERSA SS FFA ED 00789R002200400004 Fa" 1 To see whether the mean z score might be related to the number of experiments each investigator ran, we performed a correlation between N and MEAN Z (in Table 3). Results were as follows: Correlation r-squared t(21) | p -0.0185 0.0003 -0.0941 0.9257 In summary, taking the data taken at face value (i.e. not weighted by quality analysis), we can make two statements: First, considering all available data, there do appear to be significant differences among mean z scores obtainech by different experimenters. Second, there is a nonsignificant corrélation between the number of experiments run by principal investigators and their mean z scores. So to return to the question at the beginning of this section: Do different experimenters obtain about the same results? The answer is no —- experimenters in this survey showed mean z scores ranging from -2.38 to 2.95. As to the question of whether only one or two individuals may be responsible for the overall significance, the answer is also no; 25% of the experimenters in our survey reported mean z scores beyond 2 and -2. 5. RNGs were nonrandom This criticism may be addressed by examining the results of control studies reported in the literature. The results shown in Table 4 were compiled from 14 of the 44 detailed references referred to in Table 3, and were contributed by the following twelve authors: Dunne (Princeton), 57 control studies; Schmidt, 23; Broughten, 8; Braud, 2; and one each for Bierman, Hill, May, Millar, Morris, Schechter, Honorton, and Palmer. The other references did not report control results in detail and could not be used. » Table 4: Combined control studies Number of _ Data control studies z's Zz overall z p (2-tail) Survey 41 -0.012 -0.0003 ~0.002 0.999 1.036 Princeton 0.0496 0.375 0.708 Combined 0.0287 0.285 0.776 A variance test of the observed standard deviation (sd = -905) against the expected variance of 1.0 for 98 samples results in a chi-square = 80.2645 (97 df), z = -1.22, and Pp < .222 (2-tailed). Thus, for the references where control runs were described in sufficient detail to determine the B-19 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
pproved For Release APNE DE § GERIPE§9-00789R002200400001-3 number of binary hits and trials, there is no evidence of systematic (mean or variance) bias in the RNG equipment. QUALITY ANALYSIS: A PROPOSAL In this section, we address how we plan to judge the quality of the published experiments. Quality analysis in effect adds a weighting factor to each experiment’s reported z, t, or p value, depending on the assessed quality of that experiment. To avoid making a subjective quality assessment for each experiment, criteria and associated weights can be defined such that if a criterion is met, the weight associated with that criterion is added to that experiment's overall weighting factor. Rosenthal (1984, p.46-48) describes a variety of factors one nfight want to consider when performing quality analyses, but it is clear that the choice of weighting criteria depends on the research context. For the present analysis, Table 5 shows our initial proposal for criteria and associated weights; these are explained following the table. Table 5: Weighting criteria for RNG quality analysis Criteria* Weighting factors Controls With data Without data local contro! runs 30 15 global control runs 20 10 other control/random tests 10 5 target bit oscillation 10 5 Data Integrity automatic hit/trial counters § tamper resistant equipment 3 automatic data recording 10 y Statistical Integrity re-specified analysis 10 ixed run Iengths 10 direction of effort stated 5 Subject type ordinary subjects 10 special subjects 4 experimenter as subject 2 Reporting clarity fully reported hits or trials and z 10 report of z, p, ort only 4 report of other statistics 2 “Significant” only 2 “nonsignificant” only 4 * See text for explanation of criteria. Explanation of RNG weighting criteria Controls In Table 5, a “local” control means the equipment was checked for randomness as _ Part of the experimental protocol. A typical design is to have an experimental run followed by a B-20 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3
Meta-Analysis Part 1 Approved For Release 20bPb@/baA SiS HAIESB 00789R002200400001-3 control run equivalent in all respects to the experiment run, but where the subject applies no “effort” to the task or is absent. A “global” control means the equipment (RNG, computer, etc.) was tested under the same conditions as used in the experiment, but Separate from the experimental sessions. “Other” control or randomness tests meant that some reference was made to control runs or randomness tests, but the detailed results were either (a) not in the report or (b) the explanation of the controls were referenced or related to a description in another article. The columns labeled “With data” and “Without data” show different weights assigned to control runs depending on whether actual data were reported. “Target bit oscillation” means the assigned “hit” bit alternated with each newly generated bit to counterbalance any possible RNG bias. Data Integrity The “automatic hit/trial counters” criterion is satisfied if the RNG equipment has an automated method of keeping track of hits and trials. “Automatic data recording” requires use of punched paper tape, magnetic tape, computer disk, or so on, to automatically record the data collected in the experiment. There are instances in the literature (especially reports from the early 1970's) where the automatic counter criterion is met, but not automatic data recording. “Tamper resistant equipment” requires either that the RNG was (a) in a locked laboratory and inaccessible to subjects at any time, (b) the experiment was under the immediate supervision of an experimenter, (c) the equipment had a “fail-safe” or interlock system that prevented disruption of or tampering with the data collection process, or (d) the device was a computer with software data protection such as a password, protected files, or so on. Statistical Integrity “Pre-specified analysis” means it is clear from the report that the Statistical analysis method was defined before data was collected. “Fixed run lengths” means the total number of trials was specified in advance of data collection. “Direction of effort stated” requires that it was clear whether the planned test was one-tailed or two-tailed, and what direction of “effort” subjects were to aim for during the experiment. Subject Integrity This category checked whether the subjects used in the experiment were ordinary, selected or special in some other way, or the experimenter(s). Stronger weight was applied to unselected subjects because it was felt they would have less invested in the experimental outcome and would be less likely to intentionally or unintentionally interfere with the equipment or procedures. Reporting Integrity If the report included the actual number of trials and hits, or the number of trials and a z, p, or t score, this was assigned the greatest weight. If it included only z, p, ort scores, this was assigned less weight. Report of any other statistics that we had to transform into the equivalent of z scores were assigned the lowest weight. In addition, reports consisting only of the statement “significant,” without supporting data, were assigned a weight of 2 and similarly, the statement “nonsignificant” was assigned a weight of 4. Method of calculating quality-weighted analysis The weighting factor per experiment would be calculated as follows: If the criterion was clearly present in the published report, the associated weighting factor would be added to that experiment’s B-21 UNCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200400001-3