Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5 2. Database Design The database schema that was used consists of four basic tables (people, documents, experiments, and units), and two basic relationships (author and parameter). See Figure 2 for an illustration of this schema. The units-table contains information about the lowest level of statistical analysis in a given experiment. For example, if 6 viewers Participated in 20 trials each, the database would contain 6 unit entries—one for the overall result for each viewer. Although our database management system is a relational database, our requirements were inherently hierarchical. That is, each of the documents contains several experiments, and each of our experiments contain several trials. In order to minimize the redundancy within the database, we attempted to include all pertinent information as high in the | hierarchy as possible. That is, if a parameter or condition applied to an entire experiment, we would record that data at the experiment level. If, on the other hand, the parameter varied across units within a given experiment, we made provision to record those data as a function of unit instead. The analyses of most of our experiments contain both individual and group Statistics. In order to prevent any trial from being “counted” multiple times, we required that all experiments be broken up into the “units” which represent the basic grouping of trials upon which a hypothesis was being tested. Thus, any given trial appears only once in the database yet we can reconstitute the group statistics at a later time, This approach offers two advantages. First, any arbitrary parameter which does not have an explicit slot in the database can be stored, thus providing flexibility. Second, we can distinguish between “independent variables” and “incidental variables.” The former are variables which are intentionally manipulated by the experimenter, and the latter are actually parameters which the experimenter either could not control or treated as insignificant. Some of the documents detail multiple analyses for a given experiment in order to compare and evaluate standard and new analytic techniques. For this effort, however, we required that only one analysis be recorded for each experiment, since our primary focus was to evaluate the parameters that effect psychoenergetic functioning and not to compare different evaluation techniques. In determining which analysis to enter into the database, we always chose a blind method over a post hoc method. If a choice still remained, we then always chose the , technique that had been developed first. Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5
Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5 Experiment Principal Investigator Subject ID Monitor ID —— 39 One-to-Many Parameters Relationship C+) Relationship FIGURE 2 DATABASE SCHEMA DESIGN FOR META-ANALYSIS The Appendix contains examples of the DBMS input sheets that were used to encode psychoenergetic data for the database, and the instructions that were given to analysts. They are included in the Appendix for completeness; there is no further discussion about them in this report. Cc. Statistical Methods 1. Effect Size Calculations Effect sizes were calculated for each experiment or condition using the formula given by Rosenthal:? z d=—, vn where n is the number of trials and z is the usual normalized output score. If no z score was given for an experiment, but a p value was, the z that would have given that p value was computed and used in the formula. The exception to this procedure was for experiments based on a sum -of-rank statistic. For those, a more appropriate effect size formula was used and is given by Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5
Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5 S- (R+ 1) d= 2 ; -fR?~1 12 where S is the average rank and R is the number of choices for each rank. 2. Comparisons Across Classes Experiments can be categorized in accordance with a number of specific variables (e.g., type of feedback, type of target, distance between the viewer and the target). Effect sizes can be examined within a given category and compared across categories. For each categorization, the following questions are of interest: (1) Question 1: Is there any evidence of psychoenergetic functioning within each of the individual categories? (2) Question 2: Is the level of psychoenergétic functioning constant across _ all experiments within a category? (3) Question 3: Is the level of psychoenergetic functioning constant across categories? (4) Question 4: If there are differences across categories, what is the relative size of the effect in each category? Table 1 shows the notation that is used in the formalism that answers these questions. To answer question 1, compare the average z score in each category with the standard normal tables. To answer question 2, compute k OM; Ow = > > nij(dij~ di.)*. jeijs) If effect sizes are homogeneous within categories, the distribution of Qw will be approximately X? with v = (Zmi ~ k) degrees of freedom. The hypothesis of homogeneity is rejected if Qw is large compared to the chi-square table entry with v degrees of freedom. To test for homogeneity within a single category, /, compute mi Owi = > nij(dij ~ di.)? - jz Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5
Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5 Similarly, the distribution of Qwi will be approximately X? with v = (mi - k) degrees of freedom, and can be examined as above. Table 1 DEFINITIONS AND META-ANALYSIS FORMALISM Basic Definitions k = number of categories mj; = number of experiments in category /;i= /,...,% djj= effect size for experiment j in category 4; /= 1, ..., ki j njj = number of trials in experiment j in category / zi; = 2 score for experiment j in category / n= dni = number of trials / > nidiy od Nj. Within Category / = average effect size ; Vij 2iy j rr eee — = A, Vi. = average Z score Inn i. ¥7tj & = dn. = total number of trials i oo Across Categories = overall average effect size overall average z score To answer question 3, compute k Op = > ni.(di. ~4..)*. fel If effect sizes are homogeneous across categories, the distribution of Qs will be approximately X? with v= k~/ degrees of freedom. Therefore, the hypothesis of homogeneity across categories is rejected if Qs is large compared to the appropriate entry in the chi-square table with v degrees of freedom. Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5
Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5 Finally, to answer question 4, approximate 95% confidence intervals may be computed for the average effect size within a category using i. 10 Approved For Release 2003/01/09 : CIA-RDP96-00789R002200480001-5