mabpproved Fer Release CURR SS) REF 00789R002200390001 5 number of binary hits and trials, there is no evidence of systematic (mean or variance) bias in the RNG equipment. QUALITY ANALYSIS: A PROPOSAL In this section, we address how we plan to judge the quality of the published experiments. Quality analysis in effect adds a weighting factor to each experiment’s reported z, t, or p value, depending on the assessed quality of that experiment. To avoid making a subjective quality assessment for each experiment, criteria and associated weights can be defined such that if a criterion is met, the weight associated with that criterion is added to that experiment’s overall weighting factor. Rosenthal (1984, p.46-48) describes a variety of factors one might want to consider when performing quality analyses, but it is clear that the choice of weighting criteria depends on the research context. For the present analysis, Table 5 shows our initial proposal for criteria and associated weights; these are explained following the table. Table 5: Weighting criteria for RNG quality analysis Criteria* Weighting factors With data Without data Controls local control runs 30 15 global contro] runs 20 10 other control/random tests 10 5 target bit oscillation 10 5 Data Integrity automatic hit/trial counters 5 tamper resistant equipment 5 automatic data recording 10 Statistical Integrity pre-specified analysis 10 fixed run lengths 10 direction of effort stated 5 Subject type ordinary subjects special subjects experimenter as subject Reporting clarity fully reported hits or trials and z 1 report of z, p, or t only report of other statistics “significant” only “nonsignificant” only = VASO BNN AO * See text for explanation of criteria. Explanation of RNG weighting criteria Controls In Table 5, a “local” control means the equipment was checked for randomness as part of the experimental protocol. A typical design is to have an experimental run followed by a B-20 CLASSIFIED Approved For Release onset : CIA-RDP96-00789R002200390001-5
s¥00aab0 Part 1 0 Approved For Release RP RQCYIA SSH EFDP-00789R00: 2 001-5 control run equivalent in all respects to the experiment run, but where the subject applies no “effort” to the task or is absent. A “global” control means the equipment (RNG, computer, etc.) was tested under the same conditions as used in the experiment, but separate from the experimental! sessions. “Other” control or randomness tests meant that some reference was made to control runs or randomness tests, but the detailed results were either (a) not in the report or (b) the explanation of the controls were referenced or related to a description in another article. The columns labeled “With data” and “Without data” show different weights assigned to control runs depending on whether actual data were reported. “Target bit oscillation” means the assigned “hit” bit alternated with each newly generated bit to counterbalance any possible RNG bias. Data Integrity The “automatic hit/trial counters” criterion is satisfied if the RNG equipment has an automated method of keeping track of hits and trials. “Automatic data recording” requires use of punched paper tape, magnetic tape, computer disk, or so on, to automatically record the data collected in the experiment. There are instances in the literature (especially in reports from the early 1970’s) where the automatic counter criterion is met, but not automatic data recording. “Tamper resistant equipment” requires either that the RNG was (a) in a locked laboratory and inaccessible to subjects at any time, (b) the experiment was under the immediate supervision of an experimenter, (c) the equipment had a “fail-safe” or interlock system that prevented disruption of or tampering with the data collection process, or (d) the device was a computer with software data protection such as a password, protected files, or so on. Statistical Integrity “Pre-specified analysis” means it is clear from the report that the statistical analysis method was defined before data was collected. “Fixed run lengths” means the total number of trials was specified in advance of data collection. “Direction of effort stated” requires that it was clear whether the planned test was one-tailed or two-tailed, and what direction of “effort” subjects were to aim for during the experiment. Subject Integrity This category checked whether the subjects used in the experiment were ordinary, selected or special in some other way, or the experimenter(s). Stronger weight was applied to unselected subjects because it was felt they would have less invested in the experimental outcome and would be less likely to intentionally or unintentionally interfere with the equipment or procedures. Reporting Integrity _ If the report included the actual number of trials and hits, or the number of trials and a z, p, or t score, this was assigned the greatest weight. If it included only z, p, or t scores, this was assigned less weight. Report of any other statistics that we had to transform into the equivalent of z scores were assigned the lowest weight. In addition, reports consisting only of the statement “significant,” without supporting data, were assigned a weight of 2 and similarly, the statement “nonsignificant” was assigned a weight of 4. Method of calculating quality-weighted analysis The weighting factor per experiment would be calculated as follows: If the criterion was clearly present in the published report, the associated weighting factor would be added to that experiment’s B-21 NCLASSIFIED Approved For Release 2000/08/08 : CIA-RDP96-00789R002200390001-5