Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
EVALUATION METHODS
Date
of Publication
December
1990
This
is
a Department
of Defense
Intelligence Document
prepared by the Technology Assessment and Support Office,
Directorate for Scientific and Technical
Intelligence
Defense
Intelligence Agency.
PREPARED BY
Technology Assessment and Support Office
(DT-S)
NOT RELEASABLE TO FOREIGN NATIONALS
Classified
by:
DIA/DT
Declassify:
OADR
T
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
TABLE OF CONTENTS
I.
PURPOSE
II.
BACKGROUND
III.
SCOPE
IV.
OPERATIONAL PROJECT EVALUATION METHODOLOGIES
A.
SCALE APPROACH
B.
FUZZY SET APPROACH
C.
CONCEPT ANALYSIS
D.
CONTROL GROUPS
E.
IN-GROUP CONTROLS
V.
UTILITY EVALUATION CONSIDERATIONS
VI.
EVALUATION/DATA PROBLEMS
VII.
PREREQUISITES FOR EVALUATION
A.
RESEARCH REQUIREMENTS
B.
PROJECT TYPE
C.
ROLE OF RECORDS/PROTOCOLS/PROCEDURES
BIBLIOGRAPHY
FIGURES
1.
EVALUATION SCALES
2.
DATA CATEGORIES
3.
REMOTE VIEWING DATA/TARGET EXAMPLE
4.
DATA DEVELOPMENT,
EVALUATION
AND ASSESSMENT PROCESS
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
EVALUATION METHODS
I.
<U)
PURPOSE:
(S)
The purpose
of this
initial
report
is
to provide
a
summary of methods that have
been used for evaluating
psychoenergetics
(le.
,
remote
viewing)
data
in both
a research and
appl icat ions—ori ented en vi ronment
.
II.
(U)
BACKGROUND:
(U)
When modern research into psychoenergetics
(ie.
Extrasensory Perception)
began
in
the early
1930 's at Duke
University,
NC.
,
the experiments were designed to accommodate easy
to use statistical
methods.
Consequently ,
a small
number of
forced choice targets
(e.g.
,
a set of five cards
with different
symbols)
were developed as targets.
Results from experiments
using such targets could be readily compared with those expected
from chance guessing.
If results exceeded
a preset
value
(usually
one out
of twenty)
a case
for phenomenon existence could be made,
especially if the experimental
trials
were
large
in number
(ie.
,
several
thousand)
.
(U)
Although these early statistical
methods were
convenient,
they could not be applied to evaluate results from the
remote viewing experiments
that began
in the early
1970 's.
Targets
in most remote viewing experiments are not
limited to a
small
set of possibilities;
most early remote viewing experiments
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
used natural
sites or National Geographic pictures
that could be
almost anything.
In addition,
data output was radically different
from that required for early ESP research.
Instead
of
"guesses”
as
to what card
was
the target
(usually,
a hunch
or
intuition
prompted the participant),
a remote viewing subject actually
developed pictorial
and written material.
Data from remote
viewing sessions required a
"free response" style
that by its very
nature did not fit with any clear-cut statistical
approach.
Consequently data evaluation,
as
well
as phenomenon existence
assessments,
became much more complex.
(U)
Several statistical —based methods were subsequently
developed to help
in remote viewing data assessment.
Over-all
results,
even
if statistically significant,
could not make
a
strong case for phenomenon existence due to the small
number of
trials
involved
in
a typical remote viewing series.
It
was
inherently more time consuming to perform a single remote viewing
experiment than
a card guessing series
of hundreds,
if not
thousands ,
of trials for any single participant.
(S)
Consequently ,
free response evaluations from the
research environment were aimed at assessing data uniqueness on
a
trial—to trial
basis.
These required establ ishment
of
large
(at
least
100 or more)
homogeneous targets
in
a fixed target pool
that
needed complex statistics for assessing results.
Improved
evaluation methods followed based on artificial
intelligence
approaches.
Unfortunately,
most
xf not
all
of these statistical
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
approaches were difficult
to apply
in
an operational
environment
where targets
were not homogenous and where importance of various
target elements varied from one
trial
(or project)
to another.
Worse,
not all
relevant
information required for data assessment
would be known,
and some
"ground truth" data may even have been
wrong.
(S)
Nonetheless
an evaluation procedure,
even
if subjective
in nature,
had
to
be developed initially to at
least provide some
basis for estimating
or evaluating
the results from operational
projects.
Later,
improved methods
that examined both data
accuracy and data reliability were developed based on methods used
for applications research projects.
These improved methods have
reduced,
but not eliminated,
the subjective nature of operational
project evaluations.
The accumulation
of
a large track record for
given individuals over
time,
and performing meta-analysis of this
accumulated data base,
would be needed to further
improve over-all
assessment of such remote viewing
data,
(S)
Even though workable methods
for assessing data accuracy
and reliability have been developed,
there
is yet another
consideration for operational
projects:
How useful
was the data?
This requires another set of evaluation
(ie.
,
utility assessment)
that
is customer dependent.
Results from such assessments,
unfortunately,
ranged widely due
to differences
in evaluation
criteria that were
used.
Utility assessment criteria need to be
defined
in advance
of any operational
project
in order
to minimize
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
A.
(U)
SCALE APPROACH:
(S/NF)
A subjective based value scale
of
through
was used
in
the
past;
a value
of
indicated no correlation
to
ground
truth;
a value of
indicated
a perfect
match.
Recently,
scale values
of
0 through
(with
+ ,
— ,
variations)
have been
used.
By whatever
range
of scale
values
used,
the viewers
r
raw or
summarized data
is compared
to known
information about
a
target.
The best possible
judgement
is
made concerning approximate degree
of correlation
to
"ground
truth”.
An example
of
a specific scale
evaluation approach
is shown
on figure
1.
This
0 through
evaluation scale illustrates numerical
ratings,
percentages,
and
descriptions for degree
of correlation with regard
to essential
elements
of information
(EEI)
desired for each project.
(S/NF)
Figure
lists
the major target catagories
that are
usually of interest
in any remote viewing project.
Not
all
of
these are of concern
for any given
task.
Complex targets such as
S&T facilities for example,
are generally more difficult
to
evaluate than straight forward projects which have only
or
elements
to consider.
Where possible,
major target catagories of
interest
(e.g.,
facility function)
would be specified as part of
the desired information
in advance of
the session.
However,
all
the raw data is^examined no matter what
its relative importance or
category.
This provides
a gauge of individual
strengths and
weaknesses useful
for future target/person matching.
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
EVALUATION
SCALES
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
J 1“ s.
IU Q f
oc m 5
HO
z«z
£<02>
cgfe-O
<z
tL oegm
*-2
Og 2 s
II-
ui 3 < %
2 j
£• z S 5
x a
E - ! 3
>- c
fc o
q
Etc
SZrZ
UlO
5 <
>0
hnF
vy k
s
g
V—
Pli
Approved For Release 2000/08/08
: CU&RDP96-00789R0021 00030001 -5
ESSENTIALLY;
NO
AMBIGUOUS
DATA
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Uix
< 2
Bit
< O
OC s
cc
.
Ill CC
h- HI
•_ 0 .5.
;
i
<
coZo
H-
Q.
CCO
CO
111Q
OX
CL<
CC:0o
Ui0
>t CO
d o
O d
< co
- CO
CO ^
D O
UL
-
0C H
<8
r>-
<?<c
a> -J
* " *
rr-
LLI
<O
CO
lli0
cc<
COHo
LU
“0
CQO
CO<
LUH
CO*
CO
oc
UJ
CO<
o>
LU
<a
CO
CO
CO
h-o
LU
—)
CQo
p
CO
UiH
oa<
LU Z
5 O
P O
X Q
5Pcc
O'QL
*
m
<•••.
o
CO
IU0
cc<
COzo
o
5 0
CD Z
o £
(5
L1-
Z
LU
h- d
CO
CO
-Uj>to
H I
CC *
LU
LU
£ 0
< <
J
0C
_£ O
.
H -
0 CO
UJ
<O
. co
<
CO
0zo
h-oz3
LL
<o
co>X
°ra
x" z
H X
JO
< X
LU o
Z *
LLO
i
LU
S
CO
o<
CD
CO
UI
CC
h
q>3
UL
<H<O
>-H
<Za0
CC
LU
CL
0Zo
h*<Oo
-J
’
UJX
=>
b*
=>
>-
LL f
ui>
z p
o o
H <
ujj* CO
z o
. h
0 a
z <
UJ*
d x
q>p
UL
i
Ui>
go
Ui
DC
Q_
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
-C
E
CL
B.
(U)
FUZZY SET APPROACH:
(C)
Fuzzy Set Theory
is
an objective mathematical
framework for both verbal
and visual
analysis which
is utilized
for evaluating imprecise
data.
Imprecision results from sketches
that illustrate general
shapes
or approximate spatial
relationships.
Verbal
data generally includes more content
(ie.,
analysis)
than
visual
information.
The fuzzy set theory allows
for numerical
values
to be assigned to target elements that
represent their degree
of
importance.
These numerical
values
would be assigned by consumers
(or other expert personnel)
in
advance of any project.
Numerical
estimates
are also made of
the
raw data after
the session that represents
its degree of
correlation to and importance with the intended target.
Thus,
the
remote viewing data can
then be quantified by appropriate
calculations
to determine data accuracy and reliability.
Accuracy
is defined as
the percentage
of target material
that
is described
correctly by a viewer;
reliability
is defined
to be the percentage
of the over-all
response
that correlates
to the target.
(S/NF)
Figure
3 shows
an example
of some
of the data provided
in
a recent remote viewing experiment conducted by Stanford
Research Institute
to illustrate this procedure.
In this example,
the target used for
the experiment was
a microwave generator,
support equipment,
and testing equipment.
A viewer described over
seventy functions,
objects and relationships.
Over-all
accuracy,
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
using the fuzzy set approach,
was calculated to
be about
80%.
However,
over-all
reliability was only 65%j
this indicated that
35% of the raw data had no correlation
to the
intended target.
The product
of accuracy and reliability yields
a
’’figure
of merit”
what
is also useful
for over-all
data assessment and for examining
viewer performance over
time.
Additional
details
with regard
to
this experiment,
are
in reference
no.
of
the bibliography.
In
practice,
a simplified version
of this approach can
be used to
minimize analysis
time.
C.
(U)
CONCEPT ANALYSIS:
(S/NF)
Concept analysis
is based
on analyzing data
according to the over-all
concept rather than
on smaller
bits
of
information usually Jfound
in
a remote viewer's response.
For
example,
in figure
3,
one of the responses to a target was
"a
fairly long narrow channel”.
In concept analysis,
the concept of
tube,
or possibly gun,
would be emphasized rather than breaking
apart original
words such as
long,
narrow,
or channel.
Although
this can
be
a useful
approach,
some meaningful
data may be over
looked.
This method had not been widely used,
although
it was
useful
for initiating the fuzzy set approach.
D.
(U)
CONTROL GROUPS:
(U)
In Research
& Development activities,
control
groups are often necessary to establish a data baseline to which
the results of other experiments can be compared.
Generally,
a
control group
is
a randomly selected group of people run according
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 2 >| am tone Crary tone) ° Some kid of Gud. ComBisen OT L3edoAS SG1B SECRET/NOFORN REMOTE VIEWING DATA/TARGET EXAMPLE FIGURE 3 Approved For Release 2000/08/08 : CIA-RDP96-00789R002100030001-5 DRWEL Coes wall —einre Af A ] Me ET nee” ~ A Shree ae aaa 17] BUr D chney Mi : Poser AkE,, ? ‘e ) Sets yu pe) = wk LAYOAS
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
to one set
of protocols,
while another group
is run according
to
a different set of protocols.
Both groups seek
to achieve
the
same goal.
After
a designated testing
period,
the results are
compared.
The results indicate the effectiveness
of each protocol
independently of the other.
Control
groups can also indicate the
extent
to which remote viewing data
is different from data
generated by knowledgeable experts given
the same background
information.
This can permit estimates
to be made regarding
validity of the remote viewing data based on standard statistical
procedures.
E.
<U>
IN-GROUP CONTROLS:
(U)
A simple method often used
in
the research
community involves the use
of
a comparative approach.
In this
method,
the raw data from a session
is compared to one
of several
possible targets,
one of which
is the correct one.
Judges blind
to the actual
target attempt to make the best match.
If they
succeed,
a case can
be made for remote viewing success.
Statistics are straight forward.
However,
due to low numbers of
targets generally used for this comparison
(usually
to
6)
,
statistical strength
is quite
low.
This method
is useful,
however,
and can provide insight into the remote viewing process.
It does,
however,
minimize the significance of highly unique data
elements and
is not
a good indication of data usefulness.
V.
(U)
UTILITY EVALUATION CONSIDERATIONS:
(S/NF)
Utility refers to how useful
remote viewing data
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
proved to be
to the consumer.
Usually some form of scale value
or
statements ranging from "very useful"
to
"of
little use" have been
provided as feedback.
These utility evaluations,
when performed,
have usually suffered from lack
of consistent evaluation criteria.
In many cases,
the criteria were not ever agreed upon
in advance
of project initiation.
This
is
an
important
issue that
will
be
addressed
in future appl ications—or i ented research.
VI.
(U)
EVALUATION/DATA PROBLEMS:
(S/NF)
Sometimes
it
is difficult
to complete evaluations
since ground truth may not
be totally known,
or possibly the raw
data may contain predictive information
of an unspecified future
time period.
In such cases,
only partial
evaluations are
possible,
and final
assessments
may require months
or
years.
Such potential
delays
in data evaluation pose serious problems for
the reviewer
(ie.
,
feedback not possible)
,
as
well
as for
the
consumer who may require timely information.
(S/NF)
Another problem is who should do the evaluating?
If
only customers evaluate the operational
data,
they may not
be
capable of observing trends
or patterns that could be useful.
If
evaluations are solely determined by individuals not involved
in
operations,
they may emphasize aspects that are not operationally
important.
A combination of both views must
be considered when
possible and implemented
in
the evaluation process.
It
is also
necessary to minimize or eliminate the role of the data procedures
in final
data evaluation since this would present
a potential
for
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
assessment biasing,
(S/NF)
Other problems can
include viewers being exposed
to
inferences and deductions by open press sources
or television
for
certain current event
projects,
A monitor or observer present
during
a remote viewing session could also pose
a problem since
he
or she could unknowingly bias the
viewer.
Biasing
the source can
be due
to subliminal
cueing.
Consequently ,
thorough records must
be kept regarding possible
target related knowledge
of those
present
in the remote viewing session.
VII.
<U>
PREREQUISITES FOR EVALUATION:
A.
(U)
RESEARCH REQUIREMENTS:
(S)
Research requirements are more stringent than
operational Requirements since proof of principle or the search
for diff icul t-to-detect variables are
involved.
Consequently
,
there
is a strong need for well-defined targets and tightly
controlled protocol
so that appropriate statistics
can be applied.
B.
(U)
PROJECT
TYPE:
(S/NF)
The various projects worked on by this office
include foreign personalities,
military related targets,
event
predictions,
as
well
as search projects
involving location
of
target personal i ties
or moving equipment.
Evaluation procedures
with regard to search are very clear cut because either the
location
is accurate
( "a hit”)
or
it
is not
( ”a miss”)
.
Therefore,
search information can be evaluated separately from
broad categorical
data.
Training results are easier
to evaluate
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
in terms
of data accuracy since targets can be easily controlled
or defined.
C *
ROLE OF RECORDS/PROTOCOLS/PROCEDURES:
(S/NF)
The role of session records
is
an extremely
important one.
These records
would include the people
involved,
information provided to the project personnel,
project
timing and
other relevant data.
Such project details are recorded and
maintained in
a permanent file or automated data bases.
Specific
protocols are also followed to insure proper records and other
procedures are followed.
A companion
report,
item
in
the
bibliography,
contains protocol and methodology details.
<C)
To further assist and improve the over-all
evaluation
process,
future projects will
be evaluated and assessed according
to the procedures illustrated on figure
4.
This flow diagram
contains all
the essential steps necessary for insuring
that
appropriate actions occur and range from task
initiation through
tmal
data assessment and feedback.
Details
will
be developed to
clarify the various roles of each major phase and to identify
guidelines for establishing uniform evaluation criteria.
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved F^orft8f®T
e? fl/66'MM •: ^9V-^£^§£4ft)?$§f8&21 00030001 -5
FIGURE
"
Approved For Rfelease 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
BIBLIOGRAPHY
1.
"Sun Streak:
Grill
Flame Project Report
(U)
, ” Defense
Intelligence Agency,
Washington
DC,
(October
19,
1983),
SECRET-NOFORN-WNINTEL-SAP.
2.
"Special
Report:
Sun Streak Evaluation
(U)
,
Defense
Intelligence Agency,
Washington
DC,
(November
1989)
,
SECRET-NOFORN-WNINTEL-SAP.
3.
Daniel Druckman and John
A.
Swets,
Editors,
"Enhancing Human
Performance
Issues,
Theories,
and Techniques”
National Academy Press
,
Washington DC
(1988),
UNCLASSIFIED.
4.
John
A.
Palmer,
Charles Honorton and Jessica Utts," Reply to
the National Research Council
Study on Parapsychology”
,
Parapsychology Association,
Inc.
,
Research Triangle Park,
NC
(1988),
UNCLASSIFIED.
5.
"Report of a Workshop on Experimental
Parapsychology”
,
International Security and Commerce Program,
Office of Technology
and Assessment,
United States Congress
(February 22,
1989),
UNCLASSIFIED.
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
6.
"Operational
Project Protocols
(U)
, " DIA Document
No.
DT-S-1006-SS,
Defense
Intelligence Agency,
Washington,
DC
(November
27,
1990),
SECRET-NOFORN-WNINTEL.
7.
"SRI Studies
in Remote Viewing
(U)
, " Science Panel
Report
(March
1,
1984),
SECRET-WNINTEL.
8.
"Data Evaluation
Issues,” Paper,
(Pending publication by
DT-S)
,
UNCLASSIFIED.
9.
Beverly
S.
Humphrey et al.,
"Fuzzy Set Applications
in Remote
Viewing Analysis”,
Final
Report—Objective
A,
Task
3 Covering the
period
1 October
1986
to 30 September
1987,
Project
1291,
SRI
International,
Menlo Park,
CA
(December
1987),
UNCLASSIFIED.
10.
Edwin
C.
May,
et al.,
"Applications of Fuzzy Sets to Rote
Viewing Analysis
(U)”,
Final Report—Objective
F,
Task
Covering
the Period
October
1987 to 30 September
1988,
Project
1291,
SRI
International,
Menlo Park,
CA
(December
1988),
SECRET-NOFORN-WNINTEL.
11.
Edwin
C.
May,
"An Application Oriented Remote Viewing
Experiment
(U)",
Final Report Covering the Period
1 May
1988
to
April
1989,
Project 2740,
SRI
International,
Menlo Park,
CA
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5
(April
1989 ),
SECRET-NOFORN.
12.
Wanda
L.
W.
Luke
et
al.
f
”A Prototype Analysis System For
Special
Remote Viewing Tasks
(U) ”,
Final Report—Task 6.0.3
Covering the Period
i October
1988
to
30 September
1989,
Project
1291,
SRI
International,
Menlo Park,
CA
(October
1989),
SECRET-NOFORN-WNINTEL.
13.
Beverly
S.
Humphrey et al .
,
"Remote Viewing Evaluation
Techniques
<U)
, ",
Final Report—Objective
A,
Task
4,
Project
1291
SRI International,
Menlo Park,
CA
(December
1986),
SECRET-NOFORN-WNINTEL.
14.
K.
Ramakrishna Rao
"The Basic Experiments
in Parapsychology,
McFarland S Company,
Inc.,
Jefferson,
NC,
and London
(1984),
UNCLASSIFIED.
Approved For Release 2000/08/08
: CIA-RDP96-00789R0021 00030001 -5