Approved For Release 2000/08/10
: fig|gflj^6-00787R0001 00260003-2
APPENDIX A.
SUMMARY AND EVALUATION OF MORRIS
(1972)
FREE-RESPONSE ANALYSIS TECHNIQUE
Several techniques have been devised
to permit evaluation
of
the
correspondence between stimuli
(i.e.
,
targets)
and responses
(i.e.,
transcripts)
to estimate
the extent
to which any given transcript
is
descriptive of
any given target.
Previous researchers have derived
parametric statistical techniques
for
such,
based upon the likelihood
that
a given number
of matches
of
transcripts
to
targets would occur by
chance.
Others have developed rating or confidence scales
to analyze
such data.
Most
of
these
tests
assume
independence
of matching
(i.e.,
sampling with replacement)
although Stuart
(1942)
devised
a critical ratio
test
to handle
those
cases
in which
the judge's ratings
or responses were
not completely independent.
This
independency problem is exemplified' by Stuart's example
of
a
tendency
for
a judge
to avoid assigning any transcript
a ranking of
one
for more than one target.
If
a' judge has
ranked Transcript A number one
for Target A',
Transcript
B number one
for Target
B'
,
Transcript
C number
one
for Target
C'
,
and there are four targets
and transcripts,
then he
is unlikely- to rank anything other than Transcript
D number one
for Target
D'
.
While parametric tests have been devised
to handle such dependencies,
they are distribution based and have
a small error
in them,
an error
which becomes
larger
as N becomes
small.
Morris
(1972)
offers
a general formula
for calculating
the exact
probability of
a given sum
(or
leas)
of
ranks
for
the preferential matching
approach.
Me also provides
a
table
for representative situation calculations
274
Approved For Release 2000/08/10
; C^gRpP96-00787R0001 00260003-2
Approved For Release 2000/08/10
: CIAiafiBfflfc00787R0001 00260003-2
“OTmti
Assume the procedure whereby there are four targets
(A
through D
)
and
four
transcripts
(A through D) which must be blindly matched,
and
that
the four transcripts must be ranked one through
four
for each
target.
Then
the correct ranks,
summed across
all four
targets, can vary from
to
16.
The data matrix
is
shown in Table Al.
Following
the procedural requirements,
the sum in each target
column isl+2+3+4=10.
The diagonal
(underlined)
scores are
the only ones used in
the calculation of
the summed ranks;
thus,
the sum of
ranks
in this example
is
6.
In general,
if
there are n targets
(and n transcripts)
,
the
sum of ranks
can vary
from n
to n
, with an expected
value under
the null hypothesis
of
(1 +
2 +
... + n)
.
TABLE
Al.
EXAMPLE OF PREFERENTIAL RANKS MATRIX
TARGETS
TRANSCRIPTS
AL
AL
cl
D’
A
B
l
C
D
In general,
let
s
=
the obtained sum of
the diagonal ranks,
N =
the number
of
transcripts,
n
*
the number
of
targets,
and
l =
zero
and all positive integers
not exceeding
(s
- n)/N.
Approved For Release 2000/08/10
: C|
2X-RDP96-00787R0001 00260003-2
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
-SEdREf-
s
- n
,
s
N
P
(<
s)
= —
E
E.
/!/”
i
= n
l =
,
,
v £
- Nl
-
1,
;
{ in
n
-
'
S
—
s
- n
/!/
i
= n
,
%
~
s
(-1)
£
(•;
n!
a! (n
-
il)
!
)
('(n
-
1)
! (t
- Ni
- n)\
(i
-
NIL
-
1)
)
(A3)
For
the example
In. Table Al,
this equation
is equal
to:
Prob.
(<
)
^
(~D
(jj,
| (4
_ £)!^3!(i
- 4£
-
4 )
v
,
(i
- 4&
-
1)!
.
Mot ^
*
/. n
AM-'
256
(1 +
4 + 10)
= 0.059.
(A4)
When values
of
£ and
s become large,
the calculations become laborious,
although not
complex.
Morris
(1972)
has calculated the critical values
of
s
for one-tailed p values ranging from 0.20
to
^
,
assuming
that N = n.
If n ^
IV,
the above equation
(3) must be calculated,
as
it must
for exact
probability values
or N
>
.
For
the example given in Table Al,
Morris’
table gives
a value of
.05
< p
<
. 10
, which agrees with
the obtained exact p value.
The method
is statistically sound,
although
the Morris
(1972)
tabled
values
do not permit exact p-value determination.
The diligent
researcher
would undoubtedly choose
to perform the precise calculations by using
equation
(A3)
277
Approved For Release 2000/08|£
-RDP96-00787R0001 00260003-2
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
It must be noted
that
this statistical
test
is valid only
if
the rankings
are assigned independently
for each
target.
As Morris points out
(p.
406),
the obtained p-values should be used
oi\ly
as
a rough approximation in the
case
of one judge ranking
a constant response
transcript set
to
a constant
target pool.
This caution is emphasized especially
in the
case
that
(1) N
is
six or
less,
or
(2)
the judge has previously not assigned any transcript
a rank of one
or more
than one occasion.
The first caution
(N
<
6)
does not violate
the sampling distribution of
the
statistic;
rather,
it suggests
that
a judge
is more likely
to be
influenced
by his memory of
rank of
transcripts applied
to previous targets
when
the number of
transcripts
is small.
When the number
of
transcripts
is
larger than
6, presumably
the uncertainty increases
to the extent
that
the
judge's
rankings approximate independent responses.
No data are offered
to
support
this notion.
The second caution
is simply another means
to assess
the
independence
of
the judge's rankings.
If he has not
redundantly ranked
the
same
transcript one before,
there
is evidence he
is not behaving independently,
i.e.,
ranking with replacement.
The caution seems
reasonable.
Morris further indicates
that either
(1)
or
(2)
is particularly pertinent
if more
than one-third of
the number one rankings
are correct
and therefore
contributing substantially
to
the small value of
s.
When
a single judge
and constant
target pool are used,
other statistical procedures should be
devised and used,
contrary
to current practice among researchers.
Improvements on the Method
Two general techniques are validly offered by Morris
(1972)
to solve
the nonindependence problem.
In the
first method,
separate judges might
278
Approved For Release 2000/08gfi||0^ipDP96-00787R0001 00260003-2
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
^rffnnr
utunti
be
used
for each "ranking
of
the
targets."
This wording would suggest
that
a single judge rank all targets against
a given response
transcript.
Such
a procedure would involve sequential visits
to all targets and
necessarily rely upon the judge's memory for
at
least some target details.
A better method would be
to have
a single judge rank all transcripts
for
a given
target,
thereby having no knowledge
of
the other
targets
in
the
pool or how the same set of transcripts might be ranked
for any other
target.
The second methodological improvement requires
that
a judge be given
one response transcript
and
its
target
(unknown)
plus "other similar non-
target materials which are changed from one ranking
to
the next.
That
is,
the judge might
receive Transcript A along with materials describing
Target
A'
and nontargets
E',
F',
G'
,
etc.
(Table Al)
.
If
the number
of
targets
(plus nontargets)
is
large,
then, n
> N,
but
equation
(3)
can. still
be applied.
As N becomes large
(that
is,
the number
of
targets
in an N ~ n experiment
becomes
large),
the judge's
task becomes more difficult
in the "standard"
protocol;
therefore,
it may be more practical
to increase n
than N,
and
let each judge rank on only one
target.
A good rule of
thumb,
suggested
by Morris
(1972),
might be
to not use this
exact
test when nN
<
35.
Conctus'ion
While Morris
(1972)
is an important paper,
and his
analysis technique
is followed by many researchers,
there remains
cause
for concern.
Certainly,
it
is more desirable
to calculate
the exact probability
of
a given
s
than
to
use
the
tabled value,
and
the calculation is not very complex or demanding.
Of greater importance
is
the problem of nonindependence
of rankings by
the same judge.
Most
researchers disregard
this problem,
others argue
it
279
Approved For Release 2000/08/1 {yQ|A(ftBP96-00787R0001 00260003-2
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
frrnnrT
“uEunti
away by indicating
that
some judges do
in fact rank the same transcript
"one"
on
two or more targets.
Neither
is an acceptable approach;
the
second argument merely points
out
that
independence existed
(or
a "mistake"
was realized by
the judge)
on one specific
set
of responses.
What
is
needed
is
a more thorough measure of exact probability which
takes
into
account
the degree
of nonindependence,
much
as
a covariant might be used
in parametric analysis
to remove confounded sources
of variation.
Perhaps
of
the greatest heuristic concern
in
this method
is
its partial
use
of
the data.
For
the case where n = N,
only n of
the n
data points
(ranks)
are used.
The
(
n
-
n)
unused data becomes
large
as n increases.
For example,
in
the n
-
case,
only
25% of
the rankings enter
into
the
analysis.
In the n
-
case,
only
11%
of
rankings are used!
An exact
probability method based on the correlational relationship
in
the total
data matrix should be developed.
It would potentially provide greater
sensitivity and more confidence among readers unfamiliar with
this particular
area of research.
280
Approved For Release 2000/08/10
: GL§|Sii^!00787R0001 00260003-2
Approved For Release 2000/08/10
: CIA-RQgaMDCfi7R0001 00260003-2
udLIIU.1
References
Morris,
R.
L.
An exact method for evaluating preferentially
matched
free-response material.
Journal of the American Society for
Psychical Research ,
1972,
66,
401-407.
Siegel,
S.
Nonparametric statistics.
'New York:
McGraw-Hill,
1956.
Stuart,
C.
E.
An ESP test with drawings.
Journal of Parapsychology
,
1942,
6,
20-43.
Uspensky,
J.
V.
Introduction to Mathematical Probability.
New York:
McGraw-Hill,
1937.
Approved For Release 2000/08/10 ?^IA-RDpgfiMm»7R0001 00260003-2
Approved For Release 2000/08/10
: C IA^m^W87R0001 00260003-2
iJLMLI
APPENDIX
B.
REMOTE VIEWING PROTOCOL
The following
is
a verbatim copy
of
the remote viewing protocol,
as described
in a written communication from H.
Puthoff
to
a sponsor
in late
1977.
Approved For Release 2000/08/1 offfclA-RDPSe-OCtfSyROOOl 00260003-2
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
REMOTE VIEWING PROTOCOL
With regard
to replication of
our standard remote viewing protocols,
the basic outline
is
as given in our
tutorial paper,
A Perceptual.
Channel
for Information Transfer over Kilometer Distances:
Historical
Perspective and Recent Research,
"
H.
Puthoff and
R.
Targ,
pp.
329-354, March
1976
(4 copies
enclosed).
The elements
of
the
protocol,
each of which
I address below consist of
(a)
target pool
.
selection;
(b)
subject orientation;
(c)
outbound experimenter behavior;
(d)
inbound experimenter behavior;
(e)
post-experiment feedback;
(f)
judging procedure.
(a)
Target Pool Selection :
To carry out an experimental series
of,
say ,~n experiments -with
a subject,
a list
of outdoor
targets
>>
n
.
should be prepared
in advance
by an experimenter who will not
interact
with
the subject
or
experiment
a f ter
that.
The
targets should be
chosen
to be distinctive,
but not necessarily distinct
from each other,
that
is, rather than
a collection of nondescript street corners one
should
select bridges,
towers,
fountains,
gardens,
plazas,
etc.,
so
that
a
judge could
in principle recognize
targets
on the basis
ot
correct,
but sketchy descriptions.
On
the other hand
once h^ing
.
chosen a fountain- type target,
there should be several fountain target
,
for
a bridge
target,
several bridge
targets,
etc.,
so. that you avoi
the subject strategy
of
"I had
a
tower yesterday,
so
it
can
t
be
a
tower
today."
In Set,
the subject should be
told explicitly
that
the-
targets are not chosen
to be orthogonal
to each other.
When the target list
is made,
each
target location should be written
on
a card and placed
in an envelope,
the envelopes randomized and
th n
numbered
so as
to
lose all
track
of
a key.
These should
then be stored
in
a secure safe or similar container.
With regard
to whether
a target
is replaced
in the pool after use,
there are
two ways
to
go.
The preferable one
is
to not replace
it,
but keep near-replacement statistics
by
simply having a very
arge poo
with several similar
targets,
or else replacing
a used
fountain
targ
with another
fountain
target.
The problem with straight replacemen
is
that
the subject would,
upon becoming aware
of
a mental
image
of
a
previous
target,
be biased
to reject
it
as memory.
Therefore,
even
though straight replacement makes some psychologists happy,
it
is
actually an artifact-producing procedure.
(bj
Subiect Orientation:
Before the experiment,
the subject should be
shown ^some-
previous remote viewing results with one goal in mind
to
%t across
the
Idea
that one
should,
as best as possible,
report raw
perception rather
than analysis,
since
the
former
tends
to he correct
and
the
latter
is almost always wrong.
Figures
4 and
in
tllL
paper are good
examples.
In Figure
the subject had absolutely
no
concept
of
a pedestrian overpass,
but simply saw
a pattern of receding
squares;
the
target
in Figure
6 with passable drawings ^as interprete
as
a restaurant;
even
the
correctly dimensioned
pools
of water
in
Approved For Release 2000/08/10
: CIA-RDP96-00787R0001 00260003-2
^rnrT-
Approved For Release 2000/08/10 :uPW?EJFfW?0787R0001 00260003-2
Figure
3 were misinterpreted
as purification plant pools rather than
recreational swimming pools.
Reading
of
our book Mind Reach
(Targ and
Puthoff,
Delacorte Press,
1977)
provides a good background,
as we go
into
this aspect
in much more detail
than we could afford
in our
technical papers.
They need
to "get
it"
that
a rounded piece
of blue
metal
is just
that,
and
they shouldn't
try
to
figure out whether
it
is
a .car fender before
they say anything.'
Remind
the subject
that
imagina-
tion constitutes noise in
the channel,
and
therefore
the closer he can
get
to raw uninterpreted
imagery,
the better.
To have success
in
the
above,
the best guideline we have
found
is
to
choose as subjects
individuals who are self-confident,
not afraid
to be wrong,
uninhibited,
etc.
No psychological
test we have
tried
(and we have
tried
them all)
is
as successful
as
the above subjective assessment when it comes
to
choosing subjects.
Artist
types used
to unevaluated observation are
among
the most successful.
(c) .Q.lLtbqun^^rimenter Behavior :
At
the start of an experimental
session,
the inbound and outbound experimenters and subject should
rendezvous
for
a relaxed informal chat
in
the laboratory
setting.
(The
outbound experimenter
or experimenters must not know
the
target at
this
time.)
Together
they agree on
a
time
for
the subject description
to
start.
(E.g.,
30 minutes hence— the length of
time required for
getting
to
the furthest
target
in
the pool.
This
time
is
then an
invariant for all experiments.)
The outbound experimenter
then leaves
the lab,
uses
a random number generating procedure
to obtain a number
rom
-
...(number of
targets
in pool),
obtains
the so-numbered
envelope from
the
target pool
(preferably kept by another person)
and
leaves
the premises.
(We use
a Texas
Instruments SR-51 hand calculator
which has
a random number function.)
. After driving away from the
laboratory,
he opens
the envelope
to determine
the target,
and proceeds
to
that location,
I suggest he arrange
to park and
then come upon the
target
location at exactly
the starting
time
so his view of
it
is
fresh at experiment beginning.
He
then simply pays attention
to
the
environment and
does not
let his mind wander
(especially
to another
target)
.
It appears
not.
to matter how many people comprise the
outbound
team,
provided
they don't
(1)
just pay attention
to
each
other or
(2)
scatter about.
At
the end of
the agreed-upon target
viewing
time
they return
to
the lab
(usually
15 minutes)
(d)
I^ound
Experimenter Behavior :
During
the period
that
the outbound
experimenters are enroute
to
the
target,
the inbound
experimenter and
subject have
a period
to relax and discuss
the protocols.
(Inbound
it
is best not
to have additional observers.)
The goal
of
the inbound
experimenter during
this period
is
to make
it "safe"
for
the subject
to experience
remote viewing.
This
typically includes
a
low-key pep
talk as
to how remote viewing appears
to be
a natural,
not abnormal,
function,
that many people appear
to have done
it successfully,
even
their
.first
time,
and always
the reminder
to eschew analysis and
simply render raw impressions.
Approved For Release 2000/08/1^ CIA-RDP96-00787R0001 00260003-2
.flrflnrr
utylflSK