For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Final Report
December
1982
REVISED JULY 1983
A REMOTE VIEWING EVALUATION PROTOCOL
(U)
By:
EDWIN
C. MAY
Prepared
for:
DEFENSE INTELLIGENCE AGENCY
WASHINGTON,
D.C.
20301
Attention:
DT-5A
SGI J
CONTRACT M DA908-82-C-0034
SPECIAL ACCESS PROGRAM FOR GRILL FLAME.
RESTRICT DISSEMINATION TO ONLY INDIVIDUALS WITH VERIFIED ACCESS.
CLASSIFIED BY:
DT-5A
REVIEW ON:
31 December 2002
Copy No
!,
This document consists of 48 pages.
SRI/GF-0247
333 Ravenswood Ave. • Menlo Park, California 94025
NOT RELEASABLE TO
'
(415) 326-6200 •figCffipUKMBK
•
TWX: 910-373-1246
ctnM M ATinw a
i c
ed For Release 2000/08/TOM!M=RDP96-00788R001800dsoCWf-
l8 nationals
Approved For Release 20(2|)fSKSIA
:9Sm^0788R001 800050001 ~8
CONTENTS
(U)
LIST OF
ILLUSTRATIONS
(U)
iii
LIST OF TABLES
(U)
iv
I
OBJECTIVE
(U)
II
INTRODUCTION
(U)
III
ANALYSIS PROTOCOL
(U)
IV
CONCLUSIONS
(U)
Appendix A- -EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS
(U).
.
Appendix
B-- SUMMARY OF EVALUATION TECHNIQUE
(U)
REFERENCES
(U
ii
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oWfct6i-A&ySll
,Slio788R001 800050001 -8
FIGURES
(U)
Sample RV Assessment Form
(U)
,
A-l
Transcript
(U)
A-2
Transcript
(U)
A-3
Transcript
(U)
A-4
Transcript
(U)
A-5
Target Site Hango
(U)
A-6
Target Site Fernando
de Noronha
(U)
A-7
Target Site
Inverness
(U)
A-8
Target Site Punkaharju
(U)
iii
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 200
88R001 800050001 -8
TABLES
(U)
Task-Defined Relevance Scale for Target Elements
(U)
.
.
.
.
Quality Assessment Scale
(U)
Numerical Score Conversion Table
(U)
g
A-l
Target Element Relevance Ratings for Hango
(U)
A-2
Target Element Relevance Ratings for
Inverness
(U) .....
A-3
Target Element Relevance Ratings for Fernando de
Noronha
(U)
A-4
Target Element Relevance Ratings for Punkaharju
(U)
A-5
RV Assessment Forms for Calibration of Remote Viewing
(U)
.
.
A-6
A Rank Ordering
of Weighted Averages
(U)
iv
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oWyW€U-A&SifilfeDo788R001 800050001 -8
I
OBJECTIVE
(U)
(U)
The objective of
this task was to develop an evaluation procedure
to assess
the relative quality of
a
set of different remote viewing
(RV)
responses
.
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2(ldbhKokASSllBHBD)0788R001 800050001 -8
III
ANALYSIS PROTOCOL
(U)
(U)
To quantify the analysis procedure, we have divided the task
into four separate areas:
subject response,
target/task definition,
quality assessment,
and numerical analysis.
Figure
is
a sample RV
Assessment Form that has been designed to emphasize the separation
of
the
analysis tasks.
Each
of the parts
of
the form are described below.
(U)
The subject's
response should be prepared for analysis without
any knowledge
of either the target
site or the overall
task.
The aim
of
this method of response preparation is to reduce
a possibly redundant,
rambling response to
a coherent
set
of
concepts.
To meet
this requirement
we have developed
a
set
of initial guidelines
to the conceptualization
procedure
.
(U)
A concept
is defined as
a paraphrase of
a
single idea that
has
been expressed in the RV verbal
or drawing
response.
That coherent
idea
should not
be fragmented into component parts.
For example,
a response
might
be
of the form,
"i
see
a large,
textured, gray building."
The
single
concept that expresses
this
idea
should be "large,
textured,
gray building,
rather than four separate concepts--one for each word in the phrase.
Each
concept
should be entered under the "Transcript Concept"
column in the RV
Assessment Form.
(U)
For this initial evaluation technique,
a particular concept
should be used only once in the analysis.
(Some weighting factor propor-
tional
to concept frequency could be utilized,
but,
for the initial attempt,
only unique concepts are used.)
If in the construction
of the transcript
concept
list
a concept
later in the transcript
is
a duplicate
of
an earlier
one,
it should be
so noted by placing the concept number
of the original
concept
in the "D"
column.
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oUMCiA&SlUN£Do788R001 800050001-8
FIGURE
SAMPLE
RV
ASSESSMENT
FORM
(U)
Approved For Release 20(yftll(Sel-AS5Hf
3yiQ)788R001 800050001 -8
(U)
To utilize an analysis procedure that
is capable of quantitative
assessment,
it
is necessary to define,
in advance, what
the goals
of
the
assessment
are.
In the RV Assessment Form,
columns
"Element
of
Target
1 ’
and "Relevance"
are provided to clearly define the goal
of
the analysis.
In the ideal
situation,
an RV target
should be completely specified in
advance.
A target
typically consists of
a number of
target elements,
each of which may have varying relevance with regard to the overall
RV
task.
For any given target,
an independent
list
of target elements
should
be prepared.
The selection of what constitutes
a target element
is
left
completely to the discretion
of
the task coordinator.
The target element
must
be selected with little regard to task relevance
(target
element
relevance
is accounted for later).
Because an RV target consists,
in
principle,
of an essentially infinite number of
possible elements,
discre-
tion needs
to
be exercised
in the selection process,
(U)
For each target element identified for the
site,
the task
coordinator must define
a relevance rating.
This rating allows the
coordinator to tailor the analysis
to the task requirements.
Table
shows the
scale that
is used for the target element relevance rating.
Table
(U)
TASK-DEFINED RELEVANCE SCALE FOR TARGET ELEMENTS
(U)
Rating
Relevance Scale
A target element
of trivial
interest
A target element
of minor interest
A target element
of intermediate interest
A target element
of major interest
A target element
of key interest
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 20oU(felfiLA>&SI)FlEB)788R001 800050001 -8
(U)
For each transcript
concept
on the RV Assessment Form
(Figure
1)
,
the analyst
should attempt
to find the element on the
list of target
elements that he/she considers
to be the best match.
The analyst
should
be quite
liberal
in the concept/element matching
(i.e.,
the quality of
the match should be considered
at
this point
in
the analysis).
If
he/she
is
able
to
identify
a target element
that might be considered
a
match to the given concept,
a
is placed
in the
"p"
(present)
column
on the assessment form.
If
no element
can
be identified,
a
is placed
in the
1Tp
Tt
column.
After making
a target element identification,
the
selected target element,
and
its corresponding overall relevance rating
should
be entered
in the appropriate
columns
on the assessment
form.
(U)
Having identified a corresponding target element
for each con-
cept,
it
is now appropriate
to assess the quality of the match.
The
quality assessment
is done
on the basis
of
how well
the
single concept
in question matches the selected target element.
The judgement
is
to be
made without
regard
to any other issues,
such as importance of
the
concept
to the transcript,
or importance
(relevance)
of
the target element
to
the
target.
Table
shows the quality assessment
scale that
is used for this
part
of
the analysis.
The appropriate quality score from Table
is
entered in the
f,Quality M
column on the RV Assessment Form for each concept
for which a matching target element
has been identified.
(U)
An intermediate numerical
score
is
computed for each concept
from the relevance and quality
(Tables
1 and
2)
evaluation as follows:
S
'
= P
x R
X Q
,
where P is the value
in the
"p n column
(0 or
1);
R is
the relevance
evaluation;
and
is the quality assessment.
S
;
can assume values ranging
between
0 and 25.
Table
3 demonstrates how to determine
the
final
score,
S,
for
a given value
of
S'
for each concept.
The conversion table
is
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2O(^f^f0|:^l55fpfg0)788ROO1 800050001 -8
II
INTRODUCTION
(U)
(U)
In addressing the remote viewing
(RV)
evaluation question, we
consider two separate requirements:
•
Absolute evaluation of
a single response for
a
single unknown target.
•
Relative evaluation of
a
set of responses for
a
series
of known targets.
The first
of
these
is of the most
interest
in an operational
setting.
As
part of
the RV enhancement task, we have considered this problem in two
ways.
First,
by conducting an operational RV session between two calibra-
tion RV sessions,
a tentative
a priori assessment
of
operational efficacy
can be determined.
The evaluation is made on the basis
of performance
during the calibration sessions,
and on the basis
of adherence to a pre-
determined session structure.
(U)
A second technique for an a priori
evaluation was explored as
part
of the Fiscal Year 1982 program in an audio-linguistic task.
This
task provided indications that
careful
linguistic analysis, when coupled
with technical audio analysis,
could yield an assessment
in the absence
of knowledge about the target
.
(U)
Various techniques have been used
in the past1
in an attempt
to
solve the relative evaluation problem.
The most
common of these was the
simple rank ordering
of
all responses,
as assessed against all possible
targets used in an experimental
series.
In this procedure,
a judge is
presented with n RV transcripts and
n target
sites.
His
task
is
to arrange
*
(U)
References are listed at the end
of
this report.
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2000/UW6ECIMk1iM96-00788R001 800050001 -8
(U)
the transcripts in order of
the best
to least match for each of the
n
targets.
A simple numerical counting procedure
is then used to estimate
the likelihood that
the judge's transcript/target matches are by chance
alone.
This early technique contained little systematic structure for
determining the final
order of matches.
(S)
The first
step toward systematizing the rank order judging
procedure was
to preprocess the raw data in the transcript by "concep-
tualizing" both the verbal and the pictorial responses.
Conceptualizing
a transcript requires an analyst to paraphrase the transcript into
a list
of
coherent
statements.
This concept
list
is then compared and scored
concept-by-concept
to each
of
the targets
in the experiment.
The resulting
scores are averaged for each response,
and all responses are rank-ordered
on the basis
of these
scores. 3
This improved analysis procedure was applied
to
a number of experiments within the Technology Transfer Task for INSCOM
in FY 1980.
(U)
The problem with the above technique is that there are
no guide-
lines as
to how the analyst
should paraphrase the transcript;
furthermore,
the method
in which the concepts are to be assessed against the targets
remains undefined.
The purpose of the Evaluation Task in FY 1982 was
to
identify
a procedure that
corrected these deficiencies.
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oU)lhM2tA65JlWfeDo788R001 800050001 -8
Table
(U)
QUALITY ASSESSMENT SCALE
(U)
Rating
Discrimination Scale
i
Poor description;
only one
or
two aspects
of
the material match.
Fair description;
a few aspects
of
the material match,
but
a
large ambiguity exists.
Reasonable description; many aspects
of
the material match,
but
there remains
some ambiguity.
A
Good description;
a
large number
of aspects
of
the material
matches,
but
it
is possible
to conceive
of material
that
would
be
a better match.
Excellent description;
all
or nearly all aspects
of
the
material match.
Table
(U)
NUMERICAL SCORE CONVERSION TABLE
(U)
S
=
P x
R x
Q
Score
Normalized
Score
0.00
0.35
0.71
1.07
A
A
1 .A3
1.79
2.1A
2.50
2.86
3.21
3.57
3.93
A. 29
A .6A
1A
5.00
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oUfcM5tA&S4filfiDo788R001 800050001 -8
(u)
used
to eliminate
the nonuniformly-distributed gaps
in
scoring numbers
that occur
if one
simply uses
the product
S',
Thus,
the final
score
for
each concept ranges from
0 to
a maximum of
14.
This
conversion table
is
provided
as part of
the RV Assessment Form.
(U)
If
an assessment of
an individual
concept
is required,
the final
score
for
each concept / target-element match can be related
to
the quality
assessment
scale by using
the conversions
shown
in
the
third column of
Table
and on
the assessment
sheet.
It should
be noted,
however,
that
the
integer
scores
are used
to simplify
the remaining calculations.
(U)
To determine
a final evaluation of
the complete transcript
assessed
against
a given target,
a weighted
average of concept
scores
is
computed.
To assist
in
the calculation of
the weighted average,
a
tally box score
is
provided
at
the bottom of
the RV Assessment Form.
For
each
of
the possible
scores,
0 through
14,
the number
of concepts
that
attained
that particular
score are
counted.
For example,
if
concepts
were evaluated with
a
score
of
12,
a
is
entered
in
the box below
the
score.
If
the frequency
of occurrence
of
score
is
f ^
,
then
the final
weighted
average
is
computed
by
a,' =
e
f,
VFT x s./z
f. VfT
•
J
J
j
j
j
A^
= 0.357 A^
(U)
The normalized, weighted average
score,
A^_
,
is
entered
in
the
weighted average box on
the assessment
sheet.
The weighted average
score
has been normalized
to be within
the range
< Ak
<
5.0
To
aid
in
the
interpretation of
the result,
the quality assessment scale
(Table
2)
can be used
to assess quality of
the match between
the whole RV
response
and
the given target
site.
(U)
At
this point
in
the evaluation protocol,
the following options
are available,
depending
on
the
task requirement:
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2^^|)@(^^St|l|gE)00788R001 800050001 -8
(u)
• An n X n rank ordering on the basis
of
the weighted
averages
.
• A simple
selection
of
the best match.
• A statistical evaluation on
a concept-by-concept
basis
.
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2000/03
tDP96-00788R001 800050001 -8
IV
CONCLUSIONS
(U)
(S)
A protocol has been developed
to address
the relative evaluation
portion
of
the overall
RV transcript assessment problem.
As
a demonstra-
tion of
the
technique, we provide
in Appendix A an analysis
of
a series
of
four remote viewings
that were performed
as calibrations during
operational
sessions
at DIA in December
1981.
In this
series
the remote
viewing products were
of relatively high quality,
but nonetheless require
a sensitive
technique
to differentiate because
of
the
similarity of
the
targets
and,
hence,
of
the descriptions.
(The
series was chosen primarily
for
that reason.)
Application
of
the assessment technique resulted
in
the
correct blind matching
(highest scoring
in matches versus
cross
matches)
of
three
of
the four.
(U)
Appendix
B
is
a one page,
step-by-step procedure
for
the
application
of
this evaluation technique.
(U)
The material
in this document
thus
constitutes
an instruction
manual
or protocol
for application of
a step-by-step procedure
for quan-
titative assessment
of
the relative
target/ transcript correlations
of
a
series
of
transcripts matched
into
a series
of
targets.
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2<J^^A5Stp|fOD0788R001 800050001 -8
Appendix A
EVALUATION OF FOUR COORDINATE REMOTE VIEWINGS
(U)
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2000,
DP96-00788R001 800050001 -8
(S)
On 14 December 1981, four coordinate remote viewings were
con-
ducted
as calibrations during an operational remote viewing
session tar-
geted on
client-designated sites of interest.
The calibrations and the
operational RV session were conducted at the client's organization and
were witnessed by client representatives.
These four calibrations were
chosen as
a test bed for the evaluation procedure for the following
reasons,
(1)
they were conducted in an operational
setting,
and
(2)
the
targets had many similar features,
and would thus provide
a sensitive
test
of
the protocol
.
(S)
Figures A-l through A-4 are the
transcripts that were presented
to the analyst.
They are exactly as they were when collected,
except
that
the coordinates have been removed.
Figures A-5 through A- 8 are the
National
Geographic magazine targets that were used during the calibration
sessions.
Finally,
the task coordinator provided Tables A-l through A-4
as target element relevance scales for the four targets
in Figures A-5
through A-8
.
This completes the information that was given to the analyst,
and thus the analysis was carried out blind as to the matching target/
transcript pairs.
(U)
Table A-5
is
a compilation of the completed work
sheets that
were used by the analyst in this evaluation.
They are
shown in groups
by session number,
and alphabetized on the four targets.
(The task
coordinator first randomized the transcript order then assigned the session
number used above.)
For each of the transcripts,
the analyst
simply
included all phrases and all drawings
as concepts.
For example,
seven
concepts were found during Session
2.
(U)
All concepts were then analyzed as described in the text.
The
matching target element,
its relevance rating,
and the computed score are
shown for all possible combinations
of transcript/target pairs in Table
A-5.
The score distributions and their resulting weighted averages are
also shown in Table A-5.
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oUfcl)ElA&SFllBfio788R001 800050001 -8
^
fLj 0/C?/
0o %
Q
T i/vi
%r//ah
B
»2 •«•/<>
Q 0^
52 rueh^
aA
^
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2U<N£l8AS&HcJ£Do0788R001 800050001 -8
603.
B
5-23
u
S
-
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
1-8
Approved For Release 2(MJtilekAjS5t6HR)0788Ro^
o02
/•'3
-
- c
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2oU»^tAS&IEI€D)0788R001 800050001 -8
1 effect!
oo SX
I7>-Q C
I'D <-!?
Q citron
P> -
g
?
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2(4)bhMBtA®StFI€Oo788R001 800050001 -8
^LM^.
W^iA ^
,d-4>
’
FIGURE A-4 (b)
TRANSCRIPT 4
(concluded)
(U)
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
FIGURE
A-5
TARGET
SITE
HANGO
(U)
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
UNCLASSIFIED
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
FIGURE
A-6
TARGET
SITE
FERNANDO
de
NORONHA
(U)
Approved For Release^(|(^e|/^^||Ffp^-00788R001 800050001 -8
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 80005000
FIGURE
A-7
TARGET
SITE
INVERNESS
(U)
Approved For Release 2000/08/08 : CIA-RDP96-00788R001800050001-8 CPYRGHT UNCLASSIFIED LASSIFIEO Approved For Release NGLAS ALEK Pkc-00788R001800080001-8
Approved For Release 200
788R001 800050001 -8
Table A-l
(U)
TARGET ELEMENT RELEVANCE RATINGS FOR HANGD
(U)
Target Element
Relevance
Town
Cold
Peninsula
Rocky
Vegetation
Bay
Table A-2
(U)
TARGET ELEMENT RELEVANCE RATINGS FOR INVERNESS
(U)
Target Element
Relevance
City
River
Bridge
River banks
Vegetation
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Approved For Release 2004JbN®LAS®IH€B788R001 800050001 -8
Table A-3
(U)
TARGET ELEMENT RELEVANCE RATINGS
FOR FERNANDO DE NORONHA
(U)
Target Element
Relevance
Island
Surf
Hills
Uninhabited
Mountain peak
Temperate climate
Vegetation
Ocean
Table A-4
(U)
TARGET ELEMENT RELEVANCE RATINGS
FOR PUNKAHARJU
(U)
Target Element
Relevance
Connect
lakes
Town
Bridges
Cold
Vegetation
Islands
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-5
(continued)
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-5
(continued)
Approved For Release 2
00788R001 800050001 -8
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-5
(continued)
Approved For Release 20mMmm0788R001 800050001 -8
Nouvrrma xdiaasNvai
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-5
(continued)
Begin
Time:
15:23
Target:
Fernando
de
Noronha
Viewer:
#002
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-
(continued)
Approved For Release 20(W(M®tAS£lftfQ)788R001 800050001 -8
P
B
UNCLASSIFIED
Approved For Release 2000/08/08
: CIA-RDP96-00788R001 800050001 -8
Table
A-5
(continued)