Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Final Report for Objective D, Task 1 (RV Training Methodology) (verbatim transcription--not edited) CHRIS ZARAFONETIS Exceptions: None. Comments: It would be helpful if numbers were assigned to each protocol and its informed consent statement, and for final reports to include same as appendices. The final report should make a statement in its summary regarding any untoward events, their nature, etc.--or none during course of the study. (The Format of the report should include key points noted in the protocol--number of participants, no. and kinds of tests, etc. Recommendation: I do. 11-6-86 MICHAEL WARTELL Exceptions: None. Comments: Protocols appear to be far too complicated. Simple experiments appear to be possible using bit descriptions directly and should be pursued. Data is loosely presented and, what there is, is poorly analyzed. Additionally, procedures appear to be ineffective. Recommendation: I do. 11-6-86 PHIL SIDWELL Exceptions: This is a position paper by Gary Langford. It presents his thinking, but is not a research report. Comments: I discussed with Scott Hubbard the need to add instrumentation (such as EEG) to the RV session protocol to determine if RV activity can be seen simultaneously on an EEG, etc. Recommendation: I do not. I would keep it internal to SRI. 11-7-86 PHIL ZIMBARDO Exceptions: This is flawed report--an informal experiential analysis of the variables and processes that might affect RV Quality which at times gives the impression of being an empirical, scientific assessment of training effectiveness An outcome - evaluation study remains to be done with 49 ‘ Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 Page 6. Page 7, 8. Note: Page 21. Page 22. Page 31. “Page 37, 38. Page 39. Page 43. Page 44. Deleted in Final Deleted in Final variables between operationalized pre/post measures. Good (especially ?) data tables--not data-masking graphs. Hypnosis--use as means of noise reduction only. Minimize internal distractors through specific suggestions. Training--why not have initial training procedure use only the common RV symbols-bits (p. 7) plus some others to form a 10-unit set? This simplifies S’s task and provides a more objective basis for hit/miss feedback to subject. The bit -> Composite Analysis is a “bottom-up” perceptual process, but the Interpretive Overlay Analysis is a “top-down” Cognitive process. In everyday, non-PSI transactions with the world, humans use both. The bit-training procedure reduces to a focus on only the bottom-up procedure—-that is not how perception works. Good addition to training is to get S's to report BITS rather than holistic impressions. Question: In assessing training is there a baseline, pre-training evaluation of the S’s RV ability? There must be one in order to demonstrate it is the training per se that transformed novices into experts-~and not merely the S’s unrecognized, latent ability being tapped. Remember these are self-selected volunteers who may do so with a sense of “being intuitive.” Interpretations-—Reinforcement analysis of target feedback.... “so necessary for the functioning to be developed.” What about feedback of non-similarities? Does that weaken accurate functioning development? These data are for 1 novice and 1 expert? Need to have larger sample, especially of novices’ time-function allocation. Why reproduce Figure 2 here as Figure 7 with different captions, especially when caption for Figure 2 is more appropriate here under “Data Access?” Again-—delete this redundancy from the report. Question: Status of assertions re: the “correctness” of the “new” information or the secondary “flood” of information and re: “at this stage the viewer will be interpreting data correctly.” We need some factual basis for such generalizations. Subject Selection: Only 7 of 45 completed the training. (a) We need better information about the possible reason for failure to complete——“not wanting to devote time” is inadequate explanation. (b) Of the 7 who completed training, how many qualify as having RV ability? This is really scanty discussion of expert training. What about individual differences in style? What % of the time expert Ss are improved outcomes found with each of the two methods mentioned here. 50 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 Page 60. Hypotheses may be arrived at by the means mentioned here, but then they must be subjected to systematic evaluation by formal experiments to become “fact” or generalizations. A failure in this entire document is the confusion between the two classes of assertions. One never knows the evidential basis for most of the assertions presented. Page 65. Is it an idealized graph or data summary? If so, then we must know the N and the criterion for Quality of Viewing. Also the text confuses “practice” and “reinforcement”-—-they are separate. Page 66. This is not a “correlation.” Page 68. Not clear what events are reinforcing in the training situation. Showing someone else’s work is not per se vicarious reinforcement. It is only when the other person gets praised or appears to be happy over a good match. Page 68. Where is the data table relating “Rate of improvement and frequency of new concepts?” I do not believe that they collected RATE data. Page 72. Now we have a new, universal definition of reinforcement-—visiting a target site. Page 73. Figure 9 is hypothesized or empirical? Page 74. Ditto--also how is motivation assessed? Deleted in Finally, we get the (some) data. This must be presented earlier in the Final report. I read this as a failure of training: 4/45 novices -> advanced training. 1/4 advanced -> expert training, and 7/1 became expert? Zimbardo Comments continued: A Remote Viewing Idealized Procedure--“Outbound Targets” 1. Viewer-Subject a. Does not know 2 or 3 (below) at all. OR b. Constraints are set —- especially target categories and viewer characteristics. Cc. Response. Viewer draws scene on computer screen with a light pen and identifies descriptive features. 2. Objective Scene--Viewer-Sender a. Does not know 1 (above) or 3 below. OR b. Vary experimentally knowledge or “intimacy” with 1 (above) across viewer subject pairs. 51 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 3.  Targets/Display a. Individual set of stimuli from large pool. b. Selected randomly. c. Targets are schematic graphics of scenes with specifiable elements. d. Presented on video laser disks to 2 (above). e. While blank screen is available to 1 (above). f. A priori set of descriptors are made by ratings of each target. 4. Analysis Data set is each pair of computer-generated targets and S’s response scene which are stored in copy-protected locked files as a paired set for analysis. 5, Monitor’s a. Role must be reduced to that of handling only “logistics” of subject-viewer’s participation. (Queries, prompts must follow written standardized format-—e.g., Describe major feature in scene, or Describe context, setting, background. b. Should have no knowledge of target set in general or specifically. 6. Feedback a. Should be specifically varied within the experimental design. i. Sessions W/O feedback ” ii. Sessions with feedback after each trial within Ss. iii. Sessions with alternating feedback by stimulus sets across Ss. (e.g., Si 1st x trials = yes, 2nd x trials = no; S2 ist x trials = no, 2nd x trials = yes) b. Involves seeing the target on the video screen and also the viewer’s response. Perhaps include additional data collected at this point on S’s reactions to the hit/miss and any reasons for a miss. 7. Other Features Some viewers may be instructed to be allowed “pass” trials on which they feel their “perceptual window” is not open (e.g., a given target may have negative symbolic value that creates “noise” which adds interference to the bit-grabbing process. SCOTT HUBBARD General Response: With the exception of Dr. Sidwell, the SOC comments all appear to reflect a misunderstanding of the nature of this report. This report is not a description of experimental results; it is not a statement of protocols for 52 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewer's Comments on Interim Report for Objective E, Task 1 (Possible Anomalistic Behavior of Photon Detection System) NICK YARU Exceptions: Comments: Recommendation: HERB LEY Exceptions: Comments: Recommendation: (verbatim transcription--not edited) None. Good experiment with knowledgabie electrical isolation work. I do. 11-6-86 I need a briefing to explain the statistical procedures reflected in Tables 1 - 4. My questions re. procedures stem from my reluctance to accept (Table 2) an overall p < 0.0488 from a series in which 2 of 6 “p’s” are < 0.05 and 2 & 0.35, and 1 is an even 1.000. Is it appropriate to average p values [in this case p(av) = 0.3172]? Sequential probability of all 6 sets of data being the result of chance is, by my calculation, 2.6 x 10-8 : 1. These questions reflect my lack of understanding of methods used, I am sure. Would like to see final analysis including PMT/RV correlation. Would also need to see protocol, informed consent statement, and final review to reach a comment whether human use requirements have been met. I do not...until final PMT/RV correlation has been incorporated. MICHAEL A. WARTELL Exceptions: Comments: Recommendation: SCOTT HUBBARD Response to Ley: 11-6-86 None. None. I do. 11-6-86 The sequential probability calculation that Ley suggests appears to be the joint probability derived from multiplying the p-values listed in Table 2. (Although by my calculation, the value is 2.6 x 10-§ .) Such a 54 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 calculation confuses post-hoc analysis with @ priori prediction in the following way: If we had predicted in advance that the p-values obtained would be those observed, then a joint probability calculation would be correct. However, because no prediction about the p-values was made, we can only test the null hypothesis by examining the distribution of observed values as compared with the MCE distribution. One acceptable way to accomplish this is through the Fisher x?-test, cited in the report. An easy way to see the difference between these two approaches is to imagine a series of 10 coin tosses whose random outcome is HTTHTHHTHT. No one would claim that this outcome is extraordinary, especially because the number of heads equals the number of tails. However, applying Ley’s idea, we would multiply the p-value associated with each element (0.5) ten times. This value is 0.519 or 9.8 x 10-* | Clearly this probability would be relevant only if we had predicted the outcome of the coin toss in advance. The request to review the informed consent documents, etc., was satisfied in a separate meeting of the Human Use Review Board. Response to Press: During the verbal presentation of this experiment, Dr. Press took issue with the use of the Fisher y?-test. He believes that our method of combining p-values across sessions for a single viewer is invalid because the RV trials are no¢ statistically independent. In a follow-up letter, dated 10 November 1986, he remarks: “But the outcomes of the experiments carried out by the same individual are correlated, because that person has a certain ability, and he uses that ability in each of the experiments in which he is the subject.” We believe that Dr. Press is incorrect on this point. Our NULL - hypothesis, which is the basis of our testing, is that there is no ability. If the NULL hypothesis is true, then a series of RVs performed by the same individual is statistically equivalent to the same individual tossing a fair die~-a certain case for statistical independence. Our contention is especially true in that normal free-response biases of various forms are “normalized” out of the session p-values. 55 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Interim Report for Objective E, Task 1 NICK YARU Exceptions: Comments: Recommendation: (RV Feedback Dependency Experiment) (verbatim transcription-—-not edited) None. Appears protocols are proper to isolate feedback variable as only possible variable affecting RV performance. The issue of the information source in a correct RV response appears well thought out. I do. — 11-6-86 MICHAEL A. WARTELL Exceptions: Comments: Recommendation: BRIAN SKYRMS Exceptions: Comments: Recommendation: NEVIN LANTZ Response: None. Suggest accomplishing a two condition, simple experiment in which feedback is there or it isn’t in order to test hypothesis initially. I do. 11-6-86 None. None. I do. 11-10-86 The experiment suggested by Dr. Wartell, while appearing less complicated, is actually more difficult because of the enormous problem of entirely eliminating the feedback: it is easier to control the magnitude of the feedback than to eliminate it. Another reason for conducting the experiment as we described is to gain information about a possible interaction between routes of information transfer (real-time acquisition vs. precognition) as the magnitude of feedback varies from subliminal to supraliminal. 56 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Final Report for Objective E, Task 2 (Search) (verbatim transcription--not edited) CHRIS ZARAFONETIS Exceptions: None. Comments: Same as for “A Remote Viewing Training Methodology” (No protocol or ’ informed consent with report.) Recommendation: I do. 11-6-86 MICHAEL A. WARTELL Exceptions: None (edge effect mathematics needs closer scrutiny). Comments: “Edge effects” on statistical interpretations might be handled through a spherical display. While not quite “real world” enough, it would remove the natural edge avoidance bias. Also, why not attempt the experiments without feedback, as well. Recommendations: I do not. Question of edge effect math. 11-6-86 S. JAMES PRESS Exceptions: I do not think the analysis is sound. Comments: . I have handwritten some four pages outlining why I feel the analysis is flawed, & suggesting an alternative experiment with an appropriate analysis. Recommendation: I do not. 11-6-86. Analysis of Location of Target Material in Space and Time p-value is not computed correctly. If d = distance observed, p = P {D => d/H} = p-value. For a given individual, there are n trials (not replications), & results are correlated. There is a joint probability distribution for the n trials. Results might be d; , ..., da for a given subject. (d, , ..., da) = joint probability density function for d = (d; , ..., dn). There may also be learning with each trial, 57 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 EDWIN MAY Response: because of feedback, in which case we would expect an approximate ordering: d, = dg =... = dn. Averaging the n separate p-values ignores the trial-to-trial correlation. A problematic issue is that some d’s are more likely than others, just from the geometry of the matrix. For example, choosing the center of the matrix is likely to give smaller d’s than if the subject chooses the corners. If n subjects were to guess target position for the same fixed target position, then we could logically compare subjects. If we then changed the target position to a new location, & then had all n subjects again try their luck on that target, we could again compare subject’s ability. Doing this k times gives us an experimental procedure that can be analyzed statistically. The experiment has 2 factors: subject factor, & target factor. Analysis of variance methodology can be used in a traditional way to seek “main effects” for subject, and subject-target interaction. We can even introduce replication by having the same subject do the same target, randomly. In this event, replications are not independent, & the correlation must be accounted for. In the same experiment, subjects evaluate different targets, so direct comparisons are extremely difficult. By using p-values with changing targets in each trial (in the space condition) you are subject to the criticism often leveled at the use of p-values; namely, you can get whatever effect you like by taking a small, or a large sample, since the consistency of the test demands that even if the hypothesis is off only very slightly, we will reject the hypothesis with probability one. Thus, even if a small data set confirms the hypothesis, a larger data set will reject it! Because the subjects in the current experiment evaluate different subsets of targets (there might of course be some overlap), their output “distances” from n trials for each of them have different distributions. So what is the meaning of averaging p-values? We have examined the “edge” effects carefully with our statistical consultant, Dr. J. Utts, and found them to be both conservative and sound. Dr. Warteil’s suggestion of a spherical display, however, is quite ingenious, and we plan to look into it as a modification to the approach. In examining the analysis and the questions raised by Dr. Press, we find that he is not correct in his assessment. We plan to invite Dr. Press and Dr. Utts to SRI in early December to attempt to resolve the dispute. 58 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001 -4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Final Report for Objective E, Tasks 3 & 4 (Intuitive Data Sorting) (verbatim transcription—-not edited) HERB LEY Exceptions: None. Comments: None. Recommendation: I do. 11-7-86 FRED ZACHARIASEN Exceptions: It doesn’t seem to me that the data is really sufficiently good to distinguish the various models, either from each other or from the purely statistical prediction. Further, the model (IDS) violates such basic physical principles that it’s highly suspect on the face of it. I'd wait with this report until there’s better data. (Incidentally, no real data is shown in this report, & it should be.) Comments: None. Recommendation: I do not. 11-7-86 BRIAN SKYRMS Exceptions: None. Comments: None. Recommendation: I do. 11-7-86 ROSS ADEY Exceptions: (1) The study does not address the possible significance of individual social and cultural backgrounds in determining forced choices. (2) The concept that “information from the future is available in the present” is in the realm of one-shot learning. Differences in these processes in man and planarian worms, for example, emphasize man’s ability to utilize the armamentarium of past experience; and thus further emphasize the importance of evaluating social and cultural backgrounds. Comments: None. 59 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 Recommendation: None. 11-10-86 EDWIN MAY Response: Dr. Zachariasen is correct in that the PEAR data does not allow us to separate RA from IDS. Also, because we have not yet completed the PRNG formal experiment, he has a point with regard to the premature nature of this report. However there are a number of points that should be mentioned in favor of releasing this as an interim report. While it is true that we do not present the RNG data base as part of this report per se, we do reference a detailed analysis of the data base and describe the conclusions we drew from 330 separate experiments-—a p-value suggesting some form of a statistical anomaly of * 10-'8 for the historical data base consisting of * 109 bits. IDS provides a descriptive model of this historical data base that appears more parsimonious than does either RA or MCE. As such, it should be stated as clearly as possible in order for the model to receive proper scrutiny by our research community, and in order to provide guidelines for proper experimental protocols to test IDS correctly with RNGs. We agree with Dr. Zachariasen that our proposal appears to violate basic physical principles, but, as we say in the report, we feel that proposing an RA explanation violates even more basic physical principles. We do not present a meta-analysis of the literature claiming evidence for information flow from the future, but our proposal is based (and referenced) upon a rather large historical data base claiming that such a phenomenon is statistically “real.” It is important to note that the IDS a model is a heuristic model in that the data are best fit, assuming backward flow of information. Lastly, we do present the data from the PEAR laboratory in reduced form. Princeton provided over 23,000 individual data points consisting of 200; 2,000; 10,000; or 100,000 bits each. Data from the historical data base are also too volumnous to be included in this particular report. 60 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Final Report for Objective E, Task 5 (An RNG Experiment) (verbatim transcription--not edited) HERB LEY Exceptions: None. Comments: None. Recommendation: I do. 11-7-86 MICHAEL A. WARTELL Exceptions: None. Comments: Experimental protocols appear to be appropriate. Recommendation: I do. 11-7-86 61 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
congenic oA A A COAT TT ee Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SOC Reviewers’ Comments on Final Report for Objective E, Task 6 (An Experiment to Test Apparent RA Effects on Electrodermal Activity) (verbatim transcription--not edited) CHRIS ZARAFONETIS Exceptions: None. Comments: NOTE: A rest period was added after each effort period during actual procedure as distinct from the protocol. Was protocol amended through SRI-IRB approval? Has this been noted for record in SRI-IRB minutes? If RBC are to be used in FY 1987, the protocol and IC statement should show that blood will be drawn (how much, how many times) and there are possible complications from venipuncture, etc. How many subjects? How many men or women? Timing of hemolysis tests, anticoagulant used, etc., are important considerations as well. Recommendation: I do. 11-6-86 HERB LEY Exceptions: None. Comments: Need to see protocol, informed consent statement, for evaluation of meeting human use requirements. Pa Recommendation: I do. 11-6-86 PHIL ZIMBARDO Exceptions: The staff's abstract says: P=.08 (one-tailed) statistical test of psi superiority is reported or “very closely” approaches significance.” Jt does not; is not. If P < .05 is criterion, then any greater value is non-significant. The data table for the Page 10, Paragraph. 2 results needs to be here. The results in Page 15/16 indicate no statistical significance. No psi effect, no IDS effect. Where are data tables--at this stage we need appendices with primary data as well. Page 1 Remote Action (RA) -> causal effect Intuitive Data Sorting (IDS) -> informational effect-—viewer anticipates data change. Need to clearly parcel out hypotheses: 62 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
ae FE Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 Page 4 Pages 2 and 3 Comments: Recommendation: Hypothesis 1. RA -> influences EDA Hypothesis 2. IDS -> also occurs with EDA data Hypothesis 3. RA effects are separable from IDS and still are significant when IDS effects are parsed out. p = .08 is not significant from psi superiority. It does not “very closely approach significance” as asserted. If the statistical criterion of p < .05 is not met, there is not significance and should not be reported. Method is not clear. a. What was the EDA measure. b. How were subjects and influencer situated? Recommend: DATA ANALYSES: ANOVA Not T-test. Mixed between--within ___. 5 sources of degrees of freedom: 1. Trials--12 (for each type) Seed type-~2 Influencers--8 Subjects—-32 vA F&F WN Ww Type a influence (calm/activate)--2. Question: Were these counterbalanced in some way? Also: The dependent variable (DV) is skin resistance. 1. How is this corrected for initial B.S.R. basal levels? 2. Is there subsequent corrections? 3. Why not carry (?) our B.S.R. differences from D.V. ARTIFACT: Real time feedback was an uncontrolled variable, subject selected-—-must be controlled/included or excluded or varied (?) systematically. The derived DV masks a lot of different effects, such as arousal vs. calming influences, and should be used only after the separate analyses are presented. The SRI staff’s detection and reporting of the 2 serious artifacts in this procedure which render the results “uninterpretable” is to be commended! Their new hypothesis and suggested paradigm (pp. 7 and 8) appear viable. I do not. 11-6-86 63 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4
Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4 SCOTT HUBBARD Response to Zarafonetis: We regret that Dr. Zarafonetis received the impression that the IRB approved protocol was in some way modified or circumvented because of problems experienced by the participants. In actuality, the IRB protocol never specified the extremely detailed minute-by-minute experimental procedure. Since the influencer was asked to alternately activate and then calm the subject, the staff at MSF introduced the “rest” period as a natural time to reorient the influencer’s thinking. It was their unfamiliarity with the IDS model which prevented them from recognizing that this small element would completely disrupt the hypothesis under test. In no way was this rest period a response to any perceived stress or fatigue by the influencer or subject. The final report has been suitably revised to clarify this point. The issues surrounding planned experiments for FY 1987 will be addressed when the protocols are written, very likely in January of 1987. Response to Ley: The request to obtain the consent forms, etc., was satisfied in a separate meeting of the Human Use Review Board. Response to Zimbardo: Dr. Zimbardo is correct that a p-value of 0.08 is nonsignificant. This phrase has been deleted from the SRI overview report. The general requests for data tables that support the statistical conclusions have been forwarded to MSF for their consideration. As indicated by the title of Appendix A, the first set of results were for - the Pilot study, which was nonsignificant, as indicated on pages 15 and 16. The results cited in the SRI overview are only for the Confirmation experiment. The next series of comments appear to reflect a misunderstanding as to the basic hypothesis of the study. We proposed that MSF’s earlier work on EDA could be entirely explained by the IDS model, not that there is a mixture of RA and IDS effects. Page 1 has been rewritten to emphasize this point. In the SRI overview, we did not feel it was important to repeat the details of the EDA measurements or the physical layout of the experimental area as reported in the MSF paper. We refer Dr. Zimbardo to pp. 10 and 24 for those particular explanations. We have referred the proposed ANOVA analysis to Dr. Braud of MSF for consideration. In a telephone conversation, however, he responded that he favors a single, simple, a priori defined statistical test rather than the ANOVA “fishing expedition.” In Braud’s opinion, complex analysis that results in a statistically significant third-order term may be more confusing than revealing. 64 Approved For Release 2000/08/10 : CIA-RDP96-00789R003800420001-4