Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
however,  that  current  lists  are  inadequate  in  providing  discriminators  that  are  “fine”  enough 
both  to  describe  a   complex  target  accurately  and  to  exploit  fully  the  more  subtle  or  abstract 
information  content  of  the  RV  response.  To  decrease  the  granularity  of  the  RV  evaluation 
system,  therefore,  it  was  determined  that  the  technology  would  have  to  evolve  in  the  direction 
of  allowing  the  analyst  a   gradation  of  judgment  about  target  and  response  features,  rather  than 
the  current  “all-or-nothing”  binary  determinations.  A   preliminary  survey  of  various 
disciplines  and  their  evaluation  methods  (spanning  such  diverse  fields  as  artificial  intelligence, 
linguistics,  and  environmental  psychology)  has  revealed  a  branch  of  mathematics,  known  as 
“fuzzy  set  theory,”  which  provides  a  mathematical  framework  for  modeling  situations  that  are 
inherently  imprecise.  The  principal  architect  of  fuzzy  sets,  L.  A.  Zadeh,  has  stated: 
“One  of  the  aims  of  the  theory  of  fuzzy  sets  is  the  development  of  a   methodology 
for  the  formulation  and  solution  of  problems  which  are  too  complex  or  ill-defined 
to  be  susceptible  to  analysis  by  conventional  techniques .”s 
(U)  Because  the  task  of  RV  analysis  requires  human  judgments  about  imprecise 
situations — namely,  the  categorization  of  natural  sites  and  the  interpretation  of  abstract 
representations  of  those  sites — it  would  appear,  according  to  the  above  definition,  that  fuzzy 
set  theory  is  a   promising  line  of  inquiry,  In  the  next  section,  some  of  the  basic  concepts  of 
fuzzy  set  theory  will  be  examined,  with  the  aim  of  understanding  how  this  technology  might  be 
applied  to  the  specific  problem  of  RV  evaluation. 
2.  (U)  A   Tutorial 
(U)  In  traditional  set  theory,  an  element  is  either  a   member  of  a   set  or  it 
isn’t — e.g.,  the  number  2  is  a   member  of  the  set  of  even  numbers;  the  number  3  is  not. 
Fuzzy  set  theory  is  a   variant  of  traditional  set  theory,  in  that  it  introduces  the  concept  of 
degree  of  membership:  herein  lies  the  essence  of  its  applicability  to  the  modeling  of  imprecise 
systems.  For  example,  if  we  take  the  concept  of  age  (known  as  a   linguistic  variable  in  fuzzy 
set  parlance),  we  might  ascribe  to  it  certain  subcategories  (i.e.,  fuzzy  sets)  such  as  very  young, 
young,  middle-aged,  old,  etc.  Looking  at  very  young,  only,  as  a   fuzzy  set  example,  we  must 
Approved  For  Release  JfiH)MeiAS&lfliPJ?-00789R003800450001-1
Approved  For  Release  ^^/^/1^:^j-^>^-00789R003800450001-1 
(U) 
define  what  we  mean  by  this  concept  vis-a-vis  the  linguistic  variable  age.*  If  we  examine  the 
chronological  ages  from  1  to  30,  we  might  subjectively  assert  that  we  consider  the  ages  1 
through  4  to  represent  rather  robustly  a   spectrum  of  the  concept  very  young,  whereas  the  age 
of  30  probably  does  not  accurately  represent  very  young  at  all.  As  depicted  in  Figure  2,  fuzzy 
set  theory  allows  us  to  assign  a   numerical  value  between  0  and  1  that  represents  our  best 
subjective  estimate  as  to  how  much  each  of  the  ages  1  though  30  embodies  the  concept  very 
young. 
FIGURE  2  (U)  THE  FUZZY  SET  “VERY  YOUNG” 
(U)  Clearly,  a   different  set  of  numerical  values  would  be  assigned  to  the  ages  1 
through  30  for  the  fuzzy  sets  young,  middle-aged,  and  old — e.g.,  the  age  of  6  might  receive  a 
value  of  0.5  for  very  young,  but  a   value  of  1.0  for  young,  depending  on  context,  consensus, 
and  the  particular  application  of  the  system.  In  this  way  we  are  able  to  provide  manipulatable 
numerical  values  for  imprecise  natural  language  expressions;  in  addition,  we  are  no  longer 
forced  into  making  inaccurate  binary  decisions  such  as,  “Is  the  age  of  7  very  young  yes  or 
no?” 
(U)  It  is  important  to  note  that  the  design  of  the  fuzzy  application  occurs  in  accordance  with  the 
subjectivity  of  the  system  designer.  Fortunately,  the  fuzzy  set  technology  is  rich  enough  that  it  allows  for  a 
virtually  unrestricted  range  of  expression.  Technically  speaking,  young  is  the  fuzzy  set  and  very  is  a 
modifier,  but  it  is  beyond  the  scope  of  this  paper  to  present  terminology  in  depth. 
Approved  For  Release  20 
0789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
3.  (U)  Initial  Application  to  RV  Evaluation 
(U)  In  FY  1986,  work  began  on  an  initial  application  of  fuzzy  set  technology  to 
RV  evaluation,  which  simply  entails  an  extension  of  the  current  descriptor  list  capabilities.  In 
coding  the  targets,  an  analyst  employs  numerical  values  between  0  and  1,  inclusively,  to  rate 
each  of  the  20  descriptors  according  to  the  importance  of  its  visual  representation  in  the 
target.  For  example,  in  rating  a   National  Geographic  magazine  picture  of  the  Khyber  Pass, 
an  analyst  might  ascribe  a   value  of  0.80  for  a   “mountain”  descriptor,  a   value  of  0.20  for  a 
“desert”  descriptor,  and  values  for  other  appropriate  descriptors  in  accordance  with  their 
perceived  importance  to  the  target  as  a   whole.* 
(U)  The  rating  of  responses  is  considerably  more  subjective  than  the  rating  of 
targets.  The  analyst  is  required  to  apply  a   “confidence  rating” — i.e.,  again,  a   value  between  0 
and  1,  inclusively — as  to  what  degree  an  abstract  ideogram  is  representative  of  a   given 
descriptor.  For  example,  if  a   novice  subject  draws  a   conical-shaped  object  and  labels  it, 
“fuzzy  cone. ..wider  at  the  bottom...”  the  analyst  may  decide  that  there  is  some  justification 
for  interpreting  this  ideogram  as  a   volcano  covered  with  vegetation.  Clearly,  however,  the 
confidence  factor  for  making  this  highly  subjective  determination  is  quite  low;  the  net  result 
might  be,  therefore,  that  the  “volcano”  descriptor  might  receive  a   rating  of  0.15,  while 
“foliage”  might  receive  0.05. 
(U)  We  anticipate  that  the  primary  effect  of  implementing  this  rudimentary 
application  of  fuzzy  set  technology  will  be  to  “fine  tune”  the  figure  of  merit  scores,  such  that 
they  are  more  representative  of  the  “true”  information  content  of  an  RV  response.  The 
current  figure  of  merit  application  penalizes  certain  responses  and  inflates  others  (especially 
given  the  “noisy”  aspect  of  novice  data),  based  on  the  correctness  or  incorrectness  of  the 
analyst's  “all-or-nothing”  determination  with  regard  to  any  given  descriptor.  To  summarize, 
fuzzy  set  technology  is  attractive  in  two  important  respects:  (1)  it  affords  the  analyst  a   wider 
range  of  expression,  thereby  enabling  him/her  to  provide  a   more  realistic  portrayal  of  the 
information  contained  in  both  targets  and  responses,  and  (2)  it  is  compatible  with  the  figure 
of  merit  mathematical  formalism. 
4.  (U)  Potential  Future  Applications 
(U)  It  is  anticipated  that  the  initial  application  of  fuzzy  set  technology  to  RV 
responses  and  targets  will  greatly  enhance  the  accuracy  with  which  their  information  content  is 
*  (U)  Coding  of  both  targets  and  responses  might  be  more  “objectively”  arrived  at  via  the  consensus  of  a 
group  of  experienced  analysts. 
Approved  For  Release  2UM£rl(ASlSUtll^00789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
depicted.  A   problem  remains,  however,  with  the  inherently  large  granularity  of  the  current 
descriptor  list,  which  is  independent  of  the  potential  “fineness”  of  its  application  allowed  by 
fuzzy  set  theory — although  the  analyst  will  be  allowed  a   gradation  of  response  in  interpreting 
an  abstract  ideogram  (i.e.,  the  confidence  factors  ranging  from  0  to  1),  he/she  will  still  be 
constrained  to  interpreting  that  ideogram  according  to  20  concrete  descriptors.  These 
descriptors  are  significantly  limited  in  their  ability  both  to  portray  rich  environments  and  to 
distill  the  most  usable  information  from  abstract  RV  mentations. 
(U)  It  is  projected  that  future  descriptor  lists  will  afford  the  analyst  greater  latitude 
in  interpreting  the  more  abstract  aspects  of  RV  responses,  by  providing  basis  vector 
descriptors.  Such  descriptors  would  represent,  in  essence,  the  lowest  practicable  common 
denominator  of  abstraction  from  which  more  concrete  descriptors  might  be  generated  using 
fuzzy  set  operations  (such  as  intersection  and  union).  An  example  of  a   basis  vector  descriptor 
might  be  the  concept  of  vertical ,  which  is  an  abstraction  that  is  represented  to  varying  degrees 
in  such  concrete  descriptors  as  building ,  cliff,  mountain ,  waterfall ,  etc. 
(U)  Ultimately,  we  envision  that  evaluation  would  proceed  along  the  lines  of 
analyzing  both  the  RV  responses  and  targets  in  terms  of  fuzzy-weighted  basis  vector 
descriptors.  A   comparison  of  basis  vector  descriptors  between  responses  and  targets  could 
then  be  effected,  which  would  culminate  in  a   figure  of  merit  analysis  reflecting  the  subject's 
ability  to  debrief  the  more  abstract  components  of  the  psi  signal.  By  using  fuzzy  set 
operations,  concrete  target  and  response  descriptors  could  subsequently  be  generated  on  a 
“best  fit”  basis  from  the  basis  vector  descriptors,  and  a   figure  of  merit  evaluation  could  be 
performed  at  this  higher-order  level  also.  The  primary  benefits  of  this  type  of  procedure 
would  be  in  providing  objectification  of  abstract  response  data,  and  in  affording  more 
automated  interpretation  of  these  data  in  concrete  terms.  Furthermore,  it  would  also  be 
possible  to  track,  in  a   systematic  and  quantifiable  manner  (on  both  a   “subject-by-subject” 
and  “across  subject”  basis),  the  kinds  of  abstract  signals  that  subjects  are  receiving  reliably; 
presumably,  this  capability  might  then  be  used  to  illuminate  important  lines  of  future 
investigation  within  RV  fundamentals. 
Approved  For  Release 
00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
III  RESULTS  (U) 
(U)  The  results  of  the  FY  1986  evaluation  effort  have  been  obtained  primarily  from 
two  sources:  (1)  identification  of  inter-analyst  reliability  factors,  based  on  analysis  of  figure  of 
merit  statistics,  and  (2)  insights  into  descriptor  list  formulations  and  target  pool  composition, 
based  on  post  hoc  analysis  and  observations.  Each  of  these  areas  is  explored  in  turn  below. 
A.  (U)  Inter-Analyst  Reliability  Factors 
(U)  A   method  was  developed  in  FY  1986  for  rating  the  abilities  of  potential  remote 
viewing  analysts.  The  most  direct  method  of  accomplishing  this  was  simply  to  ask  a   candidate 
to  analyze  a   known  series  of  remote  viewings;  the  results  could  then  be  compared  to  those 
produced  by  a   proven  analyst,  374. 
(U)  Three  individuals,  432,  579,  and  642,  were  asked  separately  to  score  first  the 
targets  and  then  the  responses  used  in  a   remote  viewing  series  from  a   novice  viewer.  They 
used  a   twenty-bit  descriptor  list  (see  Table  1)  under  a   “blind”  protocol.  The  procedure 
described  in  Section  II.A.  was  used  to  calculate  figures  of  merit,  session  p-values,  and  overall 
p-values  for  each  analyst. 
(U)  Novice  remote  viewing  data,  which  have  been  collected  under  our  stimulus/response 
protocol,  contains  two  distinguishing  characteristics: 
•   The  data  tend  to  be  sparse  and  abstract. 
•   The  data  tend  to  be  noisy  (i.e.,  large  amounts  of  incorrect  information). 
(U)  If  the  descriptor  list  contains  mostly  concrete  items  rather  than  abstract  concepts 
(e.g.,  “Is  there  a   waterfall?”  versus  “Are  there  vertical  features?”),  then  an  analyst  who  is 
unwilling  or  unable  to  interpret  abstract  and/or  sparse  data  will  miss  whatever  remote  viewing 
information  may  be  present.  In  the  extreme  case,  a   literal  analyst  may  not  answer  any 
questions  on  the  descriptor  list  affirmatively.  If  it  is  assumed  that  there  was  some  remote 
viewing  information  present  in  the  abstract  response,  then  it  is  clear  that  the  literal  analyst  will 
miss  it.  As  the  responses  become  less  abstract  and  possibly  more  accurate,  the  difference 
between  an  interpretive  and  literal  analyst  becomes  less  important. 
Approved  For  Release 
00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U)  Based  upon  these  concepts,  three  hypotheses  were  formulated  that  could  be  tested 
with  the  three  candidate  analysts  listed  above: 
•   For  an  analyst  to  be  sensitive  to  novice  data,  he/she  must  be  willing  to 
interpret  abstract  data. 
•   An  interpretive  analyst  cannot  demonstrate  a   remote  viewing  effect  where 
there  is  none. 
•   For  literal  analysts,  the  difference  between  their  p-values  and  those  of 
an  interpretive  analyst  will  correlate  significantly,  on  a   session-by-session 
basis,  with  their  own  session  p-values. 
(U)  The  first  hypothesis  is  true  by  inspection.  Because  the  only  remote  viewing  output 
that  is  analyzed  is  the  one  coded  into  the  descriptor  list,  it  follows  that  an  analyst  must 
interpret  abstract  data,  or  there  are  no  data  for  analysis. 
(U)  Given  that  the  analyst  must  be  interpretive,  we  must  consider  whether  an  artifact 
could  be  induced  by  being  interpretive.  This  is  not  the  case.  Because  the  analyst  is  “blind” 
to  the  correct  target  for  a   given  session,  there  is  no  reason  to  expect  that  the  interpretation  of 
the  abstract  response  would  be  selective  in  such  a   way  as  to  match  the  intended  target  better 
than  any  other  target  in  the  series.  Because  the  probability  assessment  of  a   single  session 
involves  the  MCE  cross-matched  figure  of  merit  distribution,  any  “enhanced”  effects  are 
canceled  by  the  differential  comparison. 
(U)  As  a   result,  it  was  predicted  that  the  difference  between  the  means  of  the  actual 
figure  of  merit  and  the  MCE  (AM)  would  reflect  the  remote  viewing  information  content,  and 
that  the  difference  would  decrease  as  the  analyst  tends  to  be  more  literal.  Table  2  shows  the 
results  from  four  analysts  in  assessing  the  same  45  novice  remote  viewing  sessions.  It  should 
be  noted  that  p(AM)  is  the  probability  (derived  from  ANOVA)  of  observing  AM  under  the 
MCE  hypothesis.  Table  2  also  shows  the  probability  correlation,  [r],  described  above,  its 
degrees  of  freedom,  [df],  its  associated  probability,  [p(r)],  the  slope,  [Slope(r)],  of  the 
regression  line,  and  the  overall  p— value  achieved  by  each  analyst  for  the  series  of  45  remote 
viewings. 
Approved  For  Release  2(WMfibAS5ABI>IAo789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
Table  2 
(U)  VIEWER  NO.  454  RESULTS  CODED  BY  FOUR  ANALYSTS 
Analyst  (j) 
AM 
p(AM) 
r 
(Pj  
-   P374)  VS.  p. 
df 
P(r) 
Slope  (r) 
Overall 
Series  p 
374 
0.019 
0.441 
- 
- 
- 
0.316 
432 
0.004 
0.881 
0.421 
0.0040 
0.326 
0.702 
579 
0.007 
0.768 
0.555 
0.0001 
0.509 
0.867 
642 
-0.012 
0.573 
0.552 
0.0001 
0.558 
0.909 
UNCLASSIFIED 
(U)  All  three  analyst  candidates  produced  highly  significant  positive  correlations 
between  their  p-values  and  the  difference  between  their  p-values  and  374’s  p-value.  This 
indicates  that  the  literal  and  interpretive  analysts  will  tend  to  provide  more  similar  p-value 
estimates  as  the  quality  of  the  data  improves. 
(U)  Another  indication  of  the  difference  between  a   literal  and  interpretive  analyst  is 
AM.  Even  the  most  interpretive  analyst  did  not  find  significance  in  this  data  set — i.e., 
p(AM)  =   0.44.  Figure  3  demonstrates  the  effect  on  p-value  estimates  of  the  same  data  for  a 
literal  and  an  interpretive  analyst  from  a   AM  perspective.  An  analyst  with  a   large  and 
positive  AM  will  observe  a   larger  number  of  significant  sessions  (i.e.,  that  portion  of  the  curve 
labeled  “Matches”  above  Mkk — the  critical  value  of  the  figure  of  merit)  than  an  analyst  with  a 
small,  or  negative,  value  of  AM.  Thus,  an  optimal  way  of  selecting  analysts  is  to  choose  those 
with  larger  values  for  AM  and  smaller  values  for  Slope(r). 
Approved  For  Release  2 
00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
Interpretive  Analyst 
Literal  Analyst 
Matches 
Figure  of  Merit 
FIGURE  3  (U)  COMPARISON  BETWEEN  TYPES  OF  ANALYSTS 
B.  (U)  Response  Definition:  Descriptor  List  Formulation 
1.  (U)  Novice  Response  Descriptor  List 
(U)  A   post  hoc  examination  of  the  FY  1986  novice  RV  transcripts  has  resulted  in 
a   summary  list  of  responses  that  were  considered  by  the  analysts  to  be  the  most  troublesome 
to  interpret  within  the  highly  specific  framework  of  a   twenty-bit  descriptor  list  (see  Table  1). 
In  the  RV  training  paradigm  currently  used  by  novices,  interpretation  by  the  viewer  is  largely 
discouraged.  As  a   result,  concrete  words,  such  as  city,  lake,  tree,  or  boat,  are  often  labeled 
as  analytical  overlay  (AOL)  and  must  be  discarded  from  the  analysis  by  definition.  Abstract 
Approved  For  Release 
00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U) 
(and  less  interpretive)  descriptions,  such  as  rounded,  vertical,  and  wavy,  however,  are 
typically  encouraged  and  are  commonly  found  in  transcripts.  The  addition  of  abstract 
descriptors,  therefore,  would  enable  analysts  to  quantify  the  contents  of  a   transcript  literally, 
without  having  to  make  the  considerable  interpretive  leap  from  nebulous  responses  to  highly 
specific,  concrete  descriptors.  Table  3  summarizes  some  of  the  more  common  abstract 
descriptions  gleaned  from  novice  responses,  and  provides  suggestions  for  candidate  abstract 
descriptors  for  incorporation  into  future  lists.* 
(U)  It  has  yet  to  be  determined  how  abstract  and  concrete  descriptors  will  be 
structured  within  a   given  list — e.g.,  their  interdependence  could  be  either  hierarchical  in 
nature  or  configured  along  the  lines  of  semantic  networks.  Whatever  the  mathematical 
formalism,  it  is  anticipated  that  the  addition  of  abstract  descriptors  will  alleviate  much  of  the 
burden  of  novice  response  interpretation  for  analysts. 
2.  (U)  Advanced  Response  Descriptor  List 
(U)  An  FY  1986  experiment  consisting  of  12  outbound  sessions  was  performed  in 
which  an  advanced  remote  viewer  (No.  342)  was  permitted,  in  an  unsupervised  fashion,  to 
create  his  own  descriptor  list  in  advance  of  the  experiment.  The  viewer  was  told  only  that  the 
experiment  was  to  be  of  the  outbound  beacon  variety  using  San  Francisco  Bay  Area  targets 
and  that  his  list  should  consist  of  approximately  20  to  30  descriptors.  He  was  given  the 
novice  RV  descriptor  list  as  a   template  (see  Table  1).  The  hypothesis  under  test  was  that  the 
viewer,  himself,  would  be  most  knowledgeable  about  his  internal  perceptions  and  would 
therefore  be  most  qualified  to  objectify  these  perceptions  in  the  form  of  his  own 
“personalized”  descriptors. 
*  (U)  Table  3  is  not  meant  to  be  an  exhaustive  list  of  potential  abstract  descriptors.  It  is  merely  meant  to  be 
illustrative  by  highlighting  some  of  the  more  commonly  encountered  novice  RV  responses. 
Approved  For  Release  2M(J/08ffOTWA'-RDP9W)0789R003800450001-1
Approved  For  Release 
-00789R003800450001-1 
Table  3 
(U)  CANDIDATE  ABSTRACT  DESCRIPTORS  FOR  NOVICE  RESPONSES 
Typical  Novice  Responses 
Suggested  Abstract  Descriptor 
Category  of  Response 
Actual  Responses 
Patterns — curved, 
circular, 
rounded 
Curved,  circular,  circle,  oval, 
ellipse,  round,  wavy,  rolling, 
contours,  contoured,  sloping 
Are  patterns  of  round,  curved, 
or  circular  lines  significant  at 
the  site? 
Patterns — straight, 
angled 
Straight,  angled,  parallel, 
horizontal  lines,  verticality, 
vertical  lines,  vertical  objects, 
diagonal 
Are  patterns  of  straight,  angled, 
or  parallel  lines  significant  at 
the  site? 
Patterns —   combined 
curved  and 
straight 
Cone 
Are  a   mixture  of  curved  and 
straight  patterns  significant 
at  the  site? 
No  discernible  patterns 
Irregular,  shapeless,  uneven, 
rough,  bumpiness,  rugged 
terrain,  clusters,  irregular  blobs, 
irregular  shapes 
There  are  no  significant 
patterns  at  the  site. 
Distinct  boundaries 
Areas  of  light  and  dark, 
light  and  dark  contrast 
Distinct  boundaries  between 
light  and  dark  are  significant 
at  the  site. 
or 
Contrasting  areas  of  light  and 
dark  are  significant  at 
the  site. 
Unspecified  (generic) 
water 
Water,  wavy,  waves,  rippling, 
water  movement,  water  blue 
Is  water  a   significant  part 
of  the  scene? 
UNCLASSIFIED 
(U)  Table  4  provides  a   comparison  of  target  and  response  codings  (assigned  by  an 
analyst  on  a  blind  basis)  for  the  target  matched  with  its  correct  response  (see  Figure  4).* 
This  particular  example  was  chosen  because  our  subjective  appraisal  tells  us  that  the  quality  o 
fthe  response  does  not  seem  to  be  reflected  in  its  overall  p-value  (0.3289).  The  analyst’s 
coding  of  the  response  and  target  were  evaluated  on  a   post  hoc  basis  and  were  found  to  be 
»   (U)  The  abstract  descriptors  proposed  by  the  viewer  appear  to  hold  promise  for  codifying  some  of  the 
information  contained  in  novice  responses  (see  Table  3). 
Approved  For  Release  2oVoW^O^c?^RDpI6^0789R003800450001-1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
Table  4 
(U)  COMPARISON  OF  TARGET  VS.  RESPONSE  CODING  FOR  “BAYLANDS”  TARGET 
Bit 
Target 
Reponse 
Descriptor 
Coding* Coding* 
There  are  no  significant  patterns. 
Are  patterns  of  straight,  parallel,  or  angled  lines  significant  at  the  site? 
Are  patterns  of  round,  curved,  or  circular  lines  significant  at  the  site? 
D 
Are  a   mixture  of  round  and  straight  patterns  significant  at  the  site? 
H 
Is  a   significant  part  of  the  scene  hectic,  chaotic,  congested,  or  cluttered? 
H 
Is  a   significant  part  of  the  scene  clean,  empty,  or  open? 
Is  a   significant  part  of  the  scene  inside? 
Is  a   significant  part  of  the  scene  outside? 
Is  water  a   significant  part  of  the  scene? 
Is  sculptured  water  a   significant  part  of  the  scene  (fountains,  etc.)? 
Is  natural  water  a   significant  part  of  the  scene  (lakes,  ponds,  streams,  etc.)? 
Are  buildings  or  other  manmade  structures  a   significant  part  of  the  scene? 
Is  a   single  structure  a   significant  part  of  the  scene? 
Is/are  functional  (useful,  moving  parts,  etc.)  structure (s)  at  the  site? 
Is/are  artistic  (there  to  look  at)  structure  (s)  at  the  site? 
Is  a   single  color  predominant  at  the  scene? 
Is  foliage  a   significant  part  of  the  scene? 
Is  foliage  natural  in  appearance  at  the  scene? 
Is  foliage  significantly  sculpted,  manicured,  or  pruned  at  the  scene? 
Is  the  scene  predominantly  void  of  foliage? 
Is  motion  significantly  important  at  the  site? 
Is  ambient  noise  significant  at  the  site? 
■ 
Is  noise  generated  by  the  target? 
Is  noise  generated  by  people  adjacent  to  the  target? 
*  1  =   yes,  0  =   no 
UNCLASSIFIED 
Approved  For  Release  20 
0789R003800450001  -1
Approved  For  Release  2000/M/10^  C^-RDP|6-00789R003800450001 -1 
(u) 
reasonable — i.e.,  only  one  or  two  bit  assignments  were  arguable,  and  their  reassignment  would 
not  have  resulted  in  a   significant  p- value,  which  this  session  seemed  to  merit. 
(U)  With  analyst  error  seemingly  eliminated,  attention  was  focused  on  the  efficacy 
of  the  descriptor  list  itself.  It  was  concluded  that  the  list  was  deficient  in  its  ability  to  capture 
certain  kinds  of  information,  which  largely  accounted  for  the  perceived  accuracy  of  the 
response,  including: 
•   The  juxtaposition  of  elements  (i.e.,  spatial  relationships) 
•   The  “novelty  factor”  of  certain  elements  in  the  response 
•   The  high  specificity  of  named  (or  alluded  to)  target  elements. 
(U)  No  mechanism  exists,  as  yet,  within  descriptor  list  technology  for  capturing 
the  information  contained  in  the  spatial  relationships  between  elements.  Clearly,  as  in  the 
example  cited  in  Figure  4,  this  type  of  information  can  be  very  significant — i.e.,  the  viewer 
drew  his  response  from  the  beacon’s  actual  perspective  on  the  target.  Spatial  relationships 
appear  to  be  particularly  significant  in  advanced  RV  responses,  in  which  complex,  composite 
drawings  are  much  more  prevalent  than  in  novice  responses.  This  type  of  information  may 
eventually  be  accounted  for  by  employing  new  technologies  such  as  rule-based  expert  systems 
(see  Section  IV. B.),  which  lend  themselves  well  to  recognition  of  juxtaposed  elements. 
(U)  Another  factor  that  is  often  thought  to  be  important  both  in  novice  and 
advanced  RV  responses  is — for  want  of  a   better  term — the  novelty  or  strangeness  of  an 
element  in  a   response.  An  example  of  this  in  Figure  4  might  include  the  odd  shape  of  the 
structure’s  roof — i.e.,  a   curved  roof  is  a   slight  departure  from  normal  expectation.  The 
operative  information  here  is  embodied  in  the  idea  of  “architectural  oddity,”  a   concept  that  is 
quite  central  to  the  target  and  is  higher  in  information  content  than  what  is  expressed  by  the 
various  combinations  of  pattern  and  structure  descriptors  alone.  Another  example  is  the 
viewer’s  statement,  “...like  a   fence  present  but  not  a   fence...,”  which  is  a   somewhat  odd  and 
uncertain  phrase,  but  actually  describes  the  catwalk  guardrails  in  the  target  quite  well.  An 
experienced  analyst  might  consider  this  latter  type  of  information  to  be  qualitatively  better, 
because  it  represents  a   viewer’s  attempt  to  objectify  his  perception  without  succumbing  to  the 
pitfalls  of  analytical  overlay  (AOL).  Analyst  observations  about  response  novelty  can  be 
systematically  tested  by  devising  an  element-by-element  analysis  that  can  be  applied  across  a 
wide  qualitative  range  of  responses.  If  the  analyst  “lore”  is  correct,  it  might  be  captured 
either  by  a   “novelty”  fuzzy  weighting  factor  applied  to  descriptor  lists  or  by  expert  systems 
capabilities. 
Approved  For  Release 
00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
UNCLASSIFIED 
FIGURE  4  (U)  BAYLANDS  NATURE  INTERPRETIVE  CENTER,  WITH  RV  RESPONSE 
Approved  For  Release 
■00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
(U)  The  final  factor  that  led  to  forfeiture  of  information  entails  the  degree  of 
specificity  of  the  descriptor  list.  A   highly  detailed  response  of  advanced  RV  quality  will  suffer 
if  the  descriptor  list  quantifying  it  cannot  register  detail.  In  the  Figure  4  example,  pertinent 
(and  possibly  unique)  pieces  of  data — like  poles  being  used  in  the  structure's 
construction — are  relegated  to  relatively  nonunique  bit  categories  such  as  “patterns  of  straight, 
parallel,  or  angled  lines...”  Additional  highly  specific,  concrete  descriptors  (e.g.,  poles , 
pathways,  etc.),  therefore,  are  essential  for  descriptor  lists  quantifying  high-quality  RV  data. 
(U)  The  question  of  whether  a   viewer  is  better  able  to  devise  his  own  descriptor 
list  remains  unanswered.  Subjectively,  it  is  felt  that  analyst-derived  lists  have  tended  to  be 
more  concrete  in  nature  and  that  the  advanced  series  described  here  would  have  benefited 
from  that  kind  of  emphasis.  There  is  also  the  additional  fact  that  the  viewer  was  not  aware  of 
the  logical  consistency  rules  governing  descriptor  formulation  and  application,  and  that  an 
analyst's  awareness  of  these  procedural  mechanics  would  have  been  beneficial  to  the 
construction  of  the  list.  The  viewer’s  insights  into  abstract  descriptor  composition,  however, 
were  quite  invaluable  and  hold  important  implications  for  novice  list  construction  in  particular. 
C.  (U)  Target  Definition:  Implications  for  Target  Pool  Composition 
(U)  A   few  preliminary  guidelines  governing  target  pool  composition  have  been  distilled 
from  two  sources:  (1)  the  opinions  of  RV  monitors  about  the  appropriateness  of  various  kinds 
of  RV  targets  for  novice  viewers,  and  (2)  RV  analysts'  assessments  about  the  difficulties 
encountered  in  using  the  twenty-bit  descriptor  list  to  score  the  412  targets  currently  in  the 
novice  target  pool. 
(U)  As  a   general  rule,  the  current  subjective  consensus  is  that  targets  are  inappropriate 
for  training  purposes  if  they  exhibit  any  of  the  following  qualities: 
•   They  are  contrary  to  the  viewer’s  expectations. 
•   They  are  imbued  with  negative  emotional  impact.* 
•   They  violate  the  “spirit”  of  the  descriptor  list’s  intended  use. 
*(U)  Laboratory  anecdotal  evidence  suggests  that  targets  having  negative  emotional  impact  often  result  in 
psi-missing  responses. 
Approved  For  Release  2 
00789R003800450001  -1
Approved  For  Release  2000/^j1^^^-pj)m-00789R003800450001-1 
(U)  A   wide  range  of  target  types  was  used  in  the  FY  1986  novice  RV  training  series 
and  have  been  subjectively  determined  on  a   post  hoc  basis  to  be  of  varying  degrees  of 
appropriateness  for  this  task.  Table  5  provides  specific  examples  of  how  the  current  novice 
target  pool  may  be  problematic,  given  the  following  assumptions:  (1)  novice  viewers  had 
anticipated  that  targets  for  this  series  would  consist  of  pictures  taken  from  National 
Geographic  Magazine  featuring  large,  outdoor,  gestalt  scenes  (e.g.  cities,  mountains,  lakes)  of 
roughly  the  same  dimensionality;  and  (2)  the  twenty-bit  descriptor  list  was  appropriate  for 
coding  targets  of  this  type  only  and  nothing  else — e.g.,  use  of  the  list  for  technical  sites  or  for 
targets  featuring  “unnatural”  expressions  of  the  bits  was  inappropriate. 
(U)  While  it  has  yet  to  be  determined  empirically  (i.e.,  by  systematically  examining 
figures  of  merit)  whether  these  target  types  are  actually  problematic,  it  is  currently  the 
subjective  opinion  of  the  evaluation  team  that  these  kinds  of  targets  would  pose  the  greatest 
difficulties  for  novice  viewers. 
Table  5 
(U)  POTENTIAL  PROBLEM  AREAS  FOR  NOVICE  TARGETS 
Target  Type 
Specific  Target  Problem 
Condition  Violated 
Viewer 
expectation 
Emotional 
impact 
Intended 
use  of  list 
Close-up  photo  of  a   small 
feature  (e.g.,  a   flower, 
tree  trunk,  etc.) 
Dimensionality 
S 
\S 
Reflections  of  rock  form- 
ations in  a   still  pool 
Unusual  perspective 
is 
IS 
Moon — off-earth  photos 
S 
ts 
Sunken  ship  ruins 
Underwater  photos 
S 
IS 
Oil  derricks 
Technical  site 
is 
is 
Whale  slaughter 
S 
S 
Black  &   white  photos 
ts  \ 
Standing  dead  trees 
(without  foliage) 
Significant  target  element 
for  which  there  was  no 
descriptor 
\S 
is 
Photos  with  people 
and/or  animals 
Significant  target  element 
for  which  there  was  no 
descriptor 
s 
Ornate  interior  of  the 
Vatican  during  a 
ceremony 
Too  complex  for 
novices 
IS 
IS 
UNCLASSIFIED 
Approved  For  Release 
■00789R003800450001  -1
Approved  For  Release  2000/08/10  :   CIA-RDP96-00789R003800450001-1 
UNCLASSIFIED 
IV  RECOMMENDATIONS  (U) 
(U)  The  results  of  the  FY  1986  evaluation  effort  have  illuminated  several  areas  of 
investigation  that  may  hold  promise  for  improving  RV  evaluation  procedures.  These  areas 
include  (1)  identification  of  new  descriptor  lists  that  more  accurately  reflect  target  and 
response  information,  (2)  implementation  of  enhancement  techniques  (e.g.,  fuzzy  set  theory) 
for  attaining  greater  accuracy  from  descriptor  lists,  (3)  systematic  examination  of  inter-analyst 
reliability  factors,  and  (4)  development  of  new  technologies  (e.g.,  expert  systems)  for 
capturing  analysts’  insights  with  greater  efficiency.  Several  parallel  approaches,  which  address 
various  aspects  of  these  areas,  have  been  targeted  for  preliminary  research  in  FY  1987. 
These  include: 
•   A   “similarity”  experiment  (proposed  by  S.  J.  P.  Spottiswoode)  in  which 
an  attempt  will  be  made  to  identify  underlying  semantic  structures  in 
remote  viewing  descriptions  of  target  materials. 
•   An  approach  using  artificial  intelligence  (AI)  techniques  (proposed  by  J. 
Vallee)  for  recognizing,  analyzing,  and  describing  target  materials. 
•   In-house  approaches  directed  at 
-   Improvement  of  existing  descriptor  lists  by  incorporating  more 
abstract  descriptors  into  novice  lists  and  more  concrete  descriptors 
into  advanced  lists. 
Implementation  of  fuzzy  set  mathematical  weighting  factors  into 
existing  descriptor  lists  in  an  attempt  to  decrease  their  granularity. 
Assessment  of  the  relative  merits  of  analyst-derived  versus 
percipient-derived  descriptor  lists. 
Identification  of  possible  percipient-specific  “ideogramic 
dictionaries,”  which  might  serve  as  prescriptive  guides  for  the  RV 
analyst. 
Development  of  mission-specific  descriptor  lists  (e.g.,  for  technical 
sites) . 
Each  of  these  approaches  is  outlined  briefly  below. 
A.  (U)  Similarity  Experiment 
(U)  According  to  the  proposal  submitted  by  consultant  S.  J.  P.  Spottiswoode,  the 
proposed  similarity  experiment  is  aimed  at  improving  existing  evaluation  techniques  through 
Approved  For  Release jjnhasskkb 00789R003800450001  -1