Approved  For  Release  2000/^g^:^^PDP96-00789R003800450001-1 
I  INTRODUCTION  (U) 
(S)  Since  the  publication  of  the  initial  remote  viewing  (RV)  effort 
at  SRI  International*1,  two  basic  questions  have  remained  in  evaluating 
remote  viewing  data: 
•   What  is  the  definition  of  the  target  site? 
•   What  is  the  definition  of  the  RV  response? 
In  the  development  of  meaningful  evaluation  procedures,  we  must  address 
these  two  questions,  whether  the  RV  task  is  a   research-oriented  one  (in 
which  the  target  pool  is  known),  or  an  intelligence-oriented  mission  (in 
which  the  target  may  not  be  known) . 
(U)  In  the  older,  IEEE-style,  outbound  experiment,  definitions  of  target  and  response 
were  particularly  difficult  to  achieve.  The  protocol  for  such  an  experiment  dictated  that  an 
experimenter  travel  to  some  randomly  chosen  location  at  a   prearranged  time;  a   viewer's  task 
was  to  describe  that  location.  In  trying  to  assess  the  quality  of  the  RV  descriptions  (in  a 
series  of  trials,  for  example),  an  analyst  visited  each  of  the  sites  and  attempted  to  match 
responses  to  them.  While  standing  at  a   site,  the  analyst  had  to  determine  not  only  the 
bounds  of  the  site,  but  also  the  site  details  that  were  to  be  included  in  the  analysis.  To  cite  a 
specific  example  using  this  protocol:  if  the  analyst  were  to  stand  in  the  middle  of  the  Golden 
Gate  Bridge,  he/she  would  have  to  determine  whether  the  buildings  of  downtown  San 
Francisco,  which  are  clearly  and  prominently  visible,  were  to  be  considered  part  of  the 
Golden  Gate  Bridge  target.  The  RV  response  to  the  Golden  Gate  Bridge  target  could  be 
equally  troublesome,  because  responses  of  this  sort  were  typically  15  pages  of  dream-like  free 
associations.  A   reasonable  description  of  the  bridge  might  be  contained  in  the  response — it 
might  be  obfuscated,  however,  by  a   large  amount  of  unrelated  material.  How  was  an  analyst 
to  approach  this  problem  of  response  definition? 
(U)  The  first  attempt  at  quantitatively  defining  an  RV  response  involved  reducing  the 
raw  transcript  to  a   series  of  declarative  statements  called  concepts,2  Initially,  it  was 
(U)  References  are  listed  in  order  of  appearance  at  the  end  of  this  report. 
SECRET 
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
SECRET 
(U) 
determined  that  a   coherent  concept  should  not  be  reduced  to  its  component  parts.  For 
example,  a   small  red  VW  car  would  be  considered  a   single  concept  rather  than  four  separate 
concepts,  small,  red,  VW,  and  car .  Once  a   transcript  had  been  “conceptualized,”  the  list  of 
concepts  constituted,  by  definition,  the  RV  response.  The  analyst  rated  the  concept  lists 
against  the  sites.  Although  this  represented  a   major  advance  over  previous  methods,  no 
attempt  was  made  to  define  the  target  site. 
(S)  During  an  FY  1982  program,  a   procedure  was  developed  to  define 
both  the  target  and  response  material.3  It  became  evident  that  before  a 
site  can  be  quantified,  the  overall  remote  viewing  goal  must  be  clearly 
defined.  If  the  goal  is  simply  to  demonstrate  the  existence  of  the  RV 
phenomena,  then  anything  that  is  perceived  at  the  site  is  important.  But 
if  the  goal  is  to  gain  information  that  is  useful  to  the  intelligence 
community,  then  specific  items  at  the  site  are  important  while  others 
remain  insignificant.  For  example,  let  us  assume  that  an  office  is  a 
hypothetical  target  and  that  a   single  computer  in  that  office  is  of 
specific  interest.  Let  us  also  assume,  hypothetically,  that  a   viewer  gives 
an  accurate  description  of  the  shape  of  the  office,  provides  the  serial 
number  of  the  typewriter,  and  gives  a   complete  description  of  the  owner  of 
the  office.  Although  this  kind  of  a   response  might  provide  excellent 
evidence  for  remote  viewing,  the  target  of  interest  (the  computer)  is 
completely  missed — this  response,  therefore,  is  of  no  interest  as 
intelligence  data.  What  is  needed  is  a   specific  technique  to  allow 
assessments  that  are  mission-oriented. 
(S)  This  report  describes  a   computerized  RV  evaluation  procedure  that 
was  initially  developed  in  FY  19844  and  has  been  expanded  and  refined  in  FY 
1986,*  In  its  current  evolution,  it  is  an  analysis  that  has  been  aimed 
primarily  at  simpler,  research-oriented  tasks  using  a   known  target  pool. 
It  is  anticipated,  however,  that  future  refinements  to  existing  procedures, 
in  addition  to  the  advances  of  proposed  new  technologies,  will  allow 
evaluation  techniques  to  begin  to  address  the  more  complex  issue  of 
operational  RV  intelligence  collection. 
*(U)  This  report  constitutes  Objective  A,  Task  4,  “Remote  Viewing  Evaluation  Techniques.” 
Approved  For  Release  2000/(S££:RE¥?DP96-00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
II  METHOD  OF  APPROACH  (U) 
A.  (U)  Figure  of  Merit  Analysis 
1.  (U)  Overview 
(U)  Current  approaches  in  evaluation  technology  have  focused  on  the  refinement 
and  extension  of  the  figure  of  merit  analysis.4  Defined  in  general  terms,  this  procedure 
generates  a   figure  of  merit  (M)  between  0  and  1,  which  provides  an  accurate  assessment  of 
an  RV  response.  The  M   is  the  product  of  the  accuracy  and  reliability  with  which  an  RV 
response  describes  its  correct  target,  as  determined  by  an  analyst's  coding  of  RV  targets  and 
responses  according  to  a   “descriptor  list.”  Table  1  provides  a   representative  example  of  such 
a   list,  which  was  used  in  an  FY  1986  novice  RV  training  program.  Each  of  the  items  in  a 
descriptor  list  requires  a   binary  decision  from  the  analyst  as  to  the  item’s  presence  or  absence 
in  each  of  the  targets  and  responses.  The  mathematical  formalism  for  converting  the  analyst’s 
binary  codes  into  Ms  and  their  controls  is  detailed  in  Section  A.2.  below. 
2.  (U)  Mathematical  Formalism 
a.  (U)  Definitions 
(U)  For  a   single  viewer,  the  overall  method  of  analysis  consists  of  calculating 
a   figure  of  merit,  M,  for  each  viewing  session,  and  then  comparing  these  Ms  to  a   control  set 
of  figures  of  merit. 
Approved  For  Release  MflW0eH^4€lW-RW96-00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
Table  1 
(U)  DESCRIPTOR-BIT  DEFINITION 
Bit 
No. 
Descriptor 
Is  any  significant  part  of  the  scene  hectic,  chaotic,  congested,  orcluttered? 
Does  a   single  major  object  or  structure  dominate  the  scene? 
B 
Is  the  central  focus  or  predominant  ambience  of  the  scene  primarily  natural 
rather  than  artificial  or  manmade? 
Do  the  effects  of  the  weather  appear  to  be  a   significant  part  of  the  scene? 
(e.g.,  as  in  the  presence  of  snow  or  ice,  evidence  of  erosion,  etc.) 
I 
Is  the  scene  predominantly  colorful,  characterized  by  a   profusion  of  color, 
by  a   strikingly  contrasting  combination  of  colors,  or  by  outstanding,  brightly- 
colored  objects  (e.g.,  flowers,  stained-glass  windows,  etc. — not  normally 
blue  sky,  green  grass,  or  usual  building  color)? 
Is  a   mountain,  hill,  or  cliff,  or  a   range  of  mountains,  hills,  or  cliffs  a   significant 
feature  of  the  scene? 
Is  a   volcano  a   significant  part  of  the  scene? 
Are  buildings  or  other  manmade  structures  a   significant  part  of  the  scene? 
Is  a   city  a   significant  part  of  the  scene? 
Is  a   town,  village,  or  isolated  settlement  or  outpost  a   significant  feature  of  the 
scene? 
Are  ruins  a   significant  part  of  the  scene? 
Is  a   large  expanse  of  water — specifically  an  ocean,  sea,  gulf,  lake,  or  bay — a 
significant  aspect  of  the  scene? 
Is  a   land/water  interface  a   significant  part  of  the  scene? 
Is  a   river,  canal,  or  channel  a   significant  part  of  the  scene? 
Is  a   waterfall  a   significant  part  of  the  scene? 
Is  a   port  or  harbor  a   significant  part  of  the  scene? 
Is  an  island  a   significant  part  of  the  scene? 
Is  a   swamp,  jungle,  marsh,  or  verdant  or  heavy  foliage  a   significant  part  of 
the  scene/ 
Is  a   flat  aspect  to  the  landscape  a   significant  part  of  the  scene? 
Is  a   desert  a   significant  part  of  the  scene,  or  is  the  scene  predominately  dry 
to  the  point  of  being  arid? 
UNCLASSIFIED 
Approved  For  Release  UNyCeLlA55=l-ilf>Ei-00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U)  Let  [n]  be  the  number  of  sessions  to  be  analyzed.  Suppose  also  that 
the  descriptor  list  contains  [m]  bits.  We  then  define  the  total  number  of  bits  in  a   specific 
response,  [k],  as 
Rk=  2   Rj* 
k  j=i  J  ’ 
where  R.  =   1  if  bit  [j]  were  answered  affirmatively  and  equal  to  0  otherwise.  Likewise 
j   k 
the  total  number  of  bits  in  a   specific  target,  [k] ,  as 
Tk  -   25  
Tjk  • 
where  T.  =   1  if  bit  [j]  were  answered  affirmatively  and  equal  to  0  otherwise, 
j   k 
(U)  The  accuracy  of  response,  [k] ,  (the  percent  of  target  [i]  that  is 
described  correctly)  is  given  by 
m   R. ,  T. , 
EJ  k  j  i 
T 
1=1  Ti 
The  reliability  of  response  [k]  (the  percent  of  response  [k]  that  was  correct)  is  given  by 
m   R   T. . 
SJ  k  j  l 
  
“   j-1  Rk 
(U)  Finally,  the  figure  of  merit  for  response  [k]  matched  against  target  [i]  is 
given  by 
(U)  The  analysis  can  be  considered  from  two  perspectives:  matches — i.e., 
the  figure  of  merit  is  calculated  by  matching  a   response  against  its  intended  target  (i.e.,  k   =   i), 
and  cross-matches — i.e.,  the  figure  of  merit  is  calculated  by  matching  a   response  against  some 
target  other  than  its  intended  one  (i.e.,  k   #   i). 
Approved  For  Release  20 
0789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
b.  (U)  Linear  Least-Squares  Analysis 
(U)  After  [n]  remote  viewing  sessions  have  been  completed  and  the  analysis 
described  above  performed  using  k   =   i,  there  are  [n]  figures  of  merit,  one  for  each  RV 
session,  in  order  of  session  number.  To  examine  if  there  are  any  systematic  variations  within 
this  data,  a   best-fit  straight  line  is  fitted  through  the  figures  of  merit  using  standard 
techniques.  If  [x]  is  the  session  number  (x=l   n),  consider  a   straight  line  defined  as 
M(x)  =   a   +   b  (x-x) 
where  a   =   M(x)  and  x   =   constant,  and  [a]  and  [b]  are  the  intercept  and  slope,  respectively. 
Suppose  there  are  [n]  pairs  of  points,  (x,Mx).  Then  the  slope,  which  is  calculated  by  a 
standard  least-squares  technique,  is  given  by 
n 
 n   n 
n^xMx  -   2*  SMX 
5  _     —   —   —     >  where  A   is  given  by 
A 
n 
 f   n   A2 
A   =   nj£x2-(jl)xj 
x=l  Vx=l  J 
The  intercept  is  given  by 
/  “   / 
a  =   a   +   b  x   , 
 where  a   is  given  by 
n   n 
Ex2Zmx  -   2x  SxMx 
X=1  X=1  x=l  x=l 
If  we  set  x   =   to  the  average  value  of  the  session  number,  then 
x=l 
and  [a]  becomes  the  average  value  of  the  figure  of  merit.  Thus 
a   =   M   . 
Approved  For  Release  2(WNfil9A55iBHUo789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U)  This  analysis  produces  the  average  figure  of  merit  for  a   series  and  an 
indicator,  the  slope,  of  learning. 
c.  (U)  Mean  Chance  Expectation 
(U)  The  calculation  for  the  mean  chance  expectation  (MCE)  must  be 
sensitive  to  a   number  of  possible  artifacts  or  confounding  factors: 
•   Viewer  variations  (e.g.,  viewers'  different  response  biases). 
•   General  knowledge  of  the  target  pool  (e.g.,  targets  are  known 
to  be  National  Geographic  magazine  photographs) . 
•   Specific  knowledge  of  the  target  pool  resulting  from 
trail-by-trial  feedback. 
•   Methodological  considerations  (e.g.,  viewers  are  asked  to 
respond  with  more  data  at  the  end  of  the  series  compared  to 
what  was  asked  of  them  at  the  beginning  of  the  series) , 
All  of  these  factors  will  affect  the  expected  average  figure  of  merit  and  any  session-to-session 
systematic  variation  that  may  be  present. 
(U)  A   method  for  determining  the  figure  of  merit  MCE,  which  requires  the 
fewest  number  of  assumptions  about  the  structure  of  the  data  or  the  response  biases,  involves 
the  cross-matching  of  all  the  responses  to  the  same  target  set  used  in  the  series  in  question. 
A   cross-match  is  defined  as  a   comparison  between  a   response  and  a   target  other  than  the  one 
used  in  the  session.  If  a   figure  of  merit  distribution  is  calculated  for  a   large  number  of 
cross-matches,  a   number  of  the  confounding  factors  listed  above  will  be  addressed.  To 
determine  the  session-to-session  dependencies  of  the  MCE,  however,  the  session  order  must 
be  preserved.  By  preserving  the  order  of  the  responses  and  by  calculating  [n]  sets  of 
cross-matches  at  a   time,  MCE  figure  of  merit,  slope,  and  intercept  distributions  can  be 
calculated. 
(U)  As  before,  let  [n]  be  the  number  of  sessions  in  a   series  for  a   single 
viewer.  Also,  let  the  order  of  the  responses,  Rk  be  preserved.  Define  [N]  as  the  number  of 
cross-match  cycles  through  the  ordered  set  of  [n]  responses.  The  MCE  calculation  proceeds 
as  follows: 
1.  Randomly  choose  a   target  order,  i   =   l,n  ,  such  that  k   ^   i 
where  k   =   1,  n   is  the  preserved  response  order. 
2.  Calculate  the  figure  of  merit  for  the  kth  response/target 
cross-matches  as 
Approved  For  Release  2WI^I<A§S4JeWIQo0789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
Mki  =   aki  x   rki  -  k   ^   i‘ 
where  a  and  r  are  the  accuracy  and  reliability  for  response  [k] 
cross-matched  against  target  [i],  respectively. 
3.  Do  step  2  above  for  all  [n]  sessions. 
4.  Calculate  a   slope  and  intercept  for  the  resulting  figures  of 
merit  by  the  linear  least-squares  analysis  described  above. 
5.  Repeat  steps  1  through  4  above  for  [N]  cycles  to  produce 
MCE  figure  of  merit,  slope,  and  intercept  distributions. 
(U)  It  is  important  to  note  that  MCE  distributions  are  generated  for  each 
viewer  and  are  not  summed  across  viewers.  Therefore,  individual  viewer  differences  in 
response  “biases”  are  accounted  for  by  definition. 
(U)  This  procedure  also  accounts  for  general  knowledge  of  the  target  pool 
by  the  viewer,  because  information  learned  by  this  method  in  a   given  session  will  not 
necessarily  be  associated  with  the  intended  target  for  that  session.  The  net  effect  of  this  type 
of  artifact  will  be  to  “bias”  the  MCE  figure  of  merit  distribution  toward  larger  values. 
(U)  Because  the  order  of  viewings  is  preserved,  any  knowledge  of  the  target 
pool  that  is  learned  by  the  viewer  as  a   result  of  trial-by-trial  feedback  is  accounted  for  in  two 
ways: 
1.  Information  resulting  from  increasing  knowledge  of  the  target 
pool  will  “bias”  the  MCE  figure  of  merit  distribution  toward 
larger  values. 
2.  Information  resulting  from  increasing  knowledge  of  the  target 
pool  as  a   function  of  session  number  will  “bias”  the  MCE 
slope  distribution  toward  larger  values. 
(U)  Similarly,  any  artifact  caused  by  methodological  considerations  as  a 
function  of  session,  will  also  “bias”  the  MCE  figure  of  merit  and  slope  distributions. 
d.  (U)  Probability  Assessment  and  Analysis 
(U)  There  are  a   number  hypotheses  that  can  be  tested  using  the  various 
MCE  distributions  described  above: 
•   An  individual  remote  viewing  is  statistically  beyond  MCE. 
Approved  For  Release  UN^A5SI-^l&5i-00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
•   The  series  generated  by  a   single  viewer  shows  statistical 
evidence  for  remote  viewing. 
•   There  is  evidence  above  MCE  for  remote  viewing  “learning.” 
•   The  mean  of  the  observed  figure  of  merit  distribution  is 
significantly  larger  than  the  mean  of  the  MCE  distribution. 
•   The  observed  figure  of  merit  distribution  is  significantly 
different  than  the  MCE  distribution. 
(U)  Using  the  MCE  figure  of  merit  distribution,  a  straightforward  calculation 
of  areas  will  determine  whether  a   particular  figure  of  merit  from  a   single  session  is  significant. 
Figure  1  shows  an  example  of  an  MCE  figure  of  merit  distribution  described  above.  Mkk  is 
the  figure  of  merit  resulting  from  session  [k].  The  probability  of  obtaining  a   figure  of  merit  of 
Mkk  or  larger  caused  by  artifact  is  the  patterned  area  divided  by  the  total  area  under  the  curve 
shown  in  Figure  1.  This  technique  can  be  used  to  assess  the  chance  likelihood  for  all  sessions 
in  a   series  by  a   single  viewer. 
Figure  of  Merit 
FIGURE  1  (U)  AN  MCE  FIGURE  OF  MERIT  DISTRIBUTION 
(U)  To  determine  whether  there  is  statistical  evidence  of  remote  viewing 
within  a   given  series  for  a   single  viewer,  the  p-values  for  the  individual  sessions  must  be 
combined.  The  primary  method  used  for  combining  p-values  was  developed  by  Fisher.5  A 
X2  with  two  degrees  of  freedom  is  computed  for  each  p-value  and  summed.  The  resulting  X2 
is  evaluated  with  2n  degrees  of  freedom  where  [n]  is  the  number  of  p-values  that  were 
combined.  If  [k]  is  the  session  number,  the  appropriate  total  X2  is  given  by 
Approved  For  Release  2(WN£lbA5SkEI€fto789R003800450001-1
Approved  For  Release  ^j^^1^^j^^^-00789R003800450001-1 
(U) 
n 
X2  =   S   
“2-°  ln  Pk-  df  =   2n 
k=l 
where  the  p   are  the  p-values  for  each  of  the  [k]  sessions.  A   second  technique  involves 
k 
testing  the  significance  of  the  average  p-value  across  all  sessions.  A   standard  z-score  is 
calculated  by 
Z   =   V 12  n   (0.5  -   p)  , 
where  p   is  the  average  p-value  and  [n]  is  number  of  sessions.6 
(U)  These  two  measures  are  sensitive  to  different  aspects  of  the  remote 
viewing  series.  For  approximately  20  or  more  sessions,  the  two  techniques  will  yield  similar 
probability  estimates  if  there  is  slight,  but  consistent  evidence  of  remote  viewing.  On  the  other 
hand,  if  there  are  a   few  very  good  results  (i.e.,  individual  p-values  <   0.001),  then  the  X2 
technique  more  accurately  reflects  the  series  as  a   whole. 
(U)  As  an  example  of  consistency,  suppose  20  sessions  having  individual 
p-values  of  0.35  each  are  analyzed.  Then  the  z-score  for  the  average  p-value  is  2.32, 
corresponding  to  a   combined  p-value  of  0.01.  The  X2  technique  yields  a   total  X2  of  42.0 
with  df  =   40,  corresponding  to  combined  p-value  of  0.40.  To  illustrate  the  X2  technique’s 
sensitivity  to  “good”  remote  viewings,  consider  the  following  p-values  for  5  individual  sessions: 
0.45,  0.72,  0.55,  0.001,  and  0.00005.  The  average  p-value  technique  yields  a   combined 
p-value  of  0.11,  while  the  X2  technique  yields  a   combined  p-value  of  approximately  0.0005. 
(U)  To  determine  whether  there  is  evidence  of  “learning”  and  whether  the 
means  of  the  actual  and  MCE  figure  of  merit  distributions  are  significantly  different,  an 
ANOVA  technique  is  used.7  By  transforming  the  data  about  the  average  value  of  the  session 
number,  the  slope  and  intercept  hypothesis  testing  may  be  done  separately.  The  F-ratios 
(from  the  ANOVA)  for  the  two  tests  are  given  below. 
Approved  For  Release  2 wMswmsR 00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
F   
(slope) 
and 
n   (b  -   b’) 2 
(n-1) 
A 
dfl  =   1;  df2  =   (n-2) 
F   
(intercept)  =       .  dfl  =   1;  df2  =   (n-2)  , 
A 
where  [a]  and  [b]  are  the  intercept  and  slope  from  the  remote  viewing  figure  of  merit  data, 
and  [a’]  and  [b’]  are  the  intercept  slope  from  the  MCE  figure  of  merit  data.  A   is  given  by 
n   2 
 n 
 n 
 n 
 n 
S^jc  +   na  +b  ^~lk2  +   2  a   b  k   -   2a  ^   M .  -   2b  y,kM, 
k=1  k=l  k=l  k=l  k=l 
A   =   
 —     
   
     . 
n-2 
(U)  Because  the  F-ratio  for  the  slope  for  the  figure  of  merit  data  is  a 
statistical  test  between  the  observed  slope  and  that  computed  from  the  MCE,  it  constitutes  an 
estimate  of  the  probability  that  remote  viewing  “learning”  occurred  over  and  above  any 
contribution  that  might  have  occurred  because  of  some  artifact.  The  F-ratio  for  the  intercept 
constitutes  an  estimate  of  the  probability  that  the  mean  of  the  figure  of  merit  distribution  is 
different  from  the  mean  of  the  MCE.  We  use  a   standard  X2  measure,  a   more  sensitive  test, 
to  determine  if  the  observed  figure  of  merit  distribution  is  statistically  identical  to  that  of  the 
MCE  distribution. 
B.  (U)  Fuzzy  Set  Theory — An  Enhancement  Technique  for  Descriptor  List 
Technology 
1.  (U)  Overview 
(U)  The  figure  of  merit  analysis  is  predicated  on  descriptor  list  technology,  which 
represents  a   significant  improvement  over  earlier  “conceptual  analysis”  techniques,  both  in 
terms  of  “objectifying”  the  analysis  of  free  response  data  and  in  increasing  the  speed  and 
efficiency  with  which  evaluation  can  be  accomplished.  It  has  become  increasingly  evident, 
Approved  For  Release  UN/QlA55ifil^R-00789R003800450001-1