Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
'
Advances in Remote-Viewing Analysis
205
sion from the jth analyst be given by xj k
. To account for analysts’
biases, the xjM were normalized by a z transformation,
where
(a, and
cr; are the mean and standard deviation of the ^th an-
alyst’s distance
scores,
Xj k . The effect of
this transformation
is
to
convert an analyst’s absolute subjective opinion to a relative one. For
the ;'th
analyst, the
largest zM indicates that the degree of corre-
spondence for response/target k
is higher than any other pair in the
series.
It does not indicate overall quality. This type of transforma-
tion was necessary since we wished to combine the assessments from
a number of different analysts.
To combine the assessments across
analysts, we computed the
mean z score for each response/target pair,
k, as:
Zk —
7 S 2?',*!
^ ’ a
J —
where Na
is the number of analysts. The number of analysts was
determined by the data. For the best response/target pair
(i.e., ses-
sion 9005, k =
5) we computed the percent change of z5 for every
additional analyst. When the addition of two new analysts produced
consecutive changes of
less than 2%,
the process was considered
complete. For
this data
set, 37 analysts were required before
this
condition was met. Figure 2 shows the normalized mean for each
target/response pair, and represents a relative assessment of remote-
viewing quality. These means
constitute the
basis for the ground
truth against which the fuzzy set technique was measured. We re-
cognize that this definition of ground truth
is based on global deci-
sions and may not be most optimal (Dawes,
1988).
Results of the Fuzzy Set Analysis
To effect a meaningful comparison between ground truth and
the figure of merit analysis, we also analyzed the same RV series
that served as the ground truth set by the fuzzy set figure of merit
method. The fuzzy set membership values
(|a’s) for the six targets
and
six responses were consensus coded by
five
analysts ranging
from expert to novice. A typical spread of )a assignments was ±0.1
with an occasional outlier. Some of the elements were vigorously de-
bated
until a consensus was reached.
Accuracies,
reliabilities, and
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
I Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
206
The Journal of Parapsychology
•
j
Session Number
Figure 2. Normalized mean for each target/response pair.
figures of merit were calculated for each target/response paii (Table
1). It should be noted that the encoding was a post hoc exercise, but
because the assignment for each element in the USE had to be de-
fended before a consensus was reached, the FMs shown in Table
constitute reasonable estimates of their “blind” equivalents. Appen-
dix B shows the target and response elements that were scored from
the universal set (see Appendix A) for Sessions 9004 and 9005. As
an example of the fuzzy calculation, Appendix B also shows the re-
Session Number
Figure 3. Comparison with ground truth.
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 000034
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
•
Advances in Remote-Viewing Analysis
207
Table
Fuzzy Set Quantities for “Ground Truth” Series
Session
Accuracy
Reliability
Figure of
merit
Rank
Fractional
rank
9001
.317
.484
.153
.403
9002
.273
.477
.130
103
.515
9003
.358
.571
.205
.155
9004
.212
.379
.080
142
.713
9005
.573
.594
.340
.015
9006
.298
.555
.165
.068
suits of the target a-cut, the fuzzy intersection, and the accuracy,
reliability, and figure of merit for Session 9005. Table
1 also shows
the absolute and relative ranks from a target pool of 200. To deter-
mine the absolute rank for each
session, we calculated
figures of
merit for
all 200 targets in the pool and placed them in numerical
order from the largest to the smallest. The absolute rank
is just the
position (from the top) of the FM corresponding to the intended
target. Ties were resolved by choosing the next larger integer rank
number to the centroid of the ties. The fractional rank number can
be considered a p value for an individual session and
is equal to the
absolute rank/200. Using Equation
1, the
overall p value for the
combined six trials
is .052 (N =
200, K =
372, m =
6). Using the
approximation (Equation
2), we compute
z =
1.633, p
.05,
to
demonstrate that for six trials, the approximation in reasonable. For
completeness, we compute the effect size (r =
0.67).
To compare the results of the fuzzy set analysis with those of the
ground truth, we linearly renormalized the ground truth figures to
be within the interval [0,1] and to possess the same maximum and
minimum. As can be seen from Figure 3, the results from the fuzzy
set analysis system parallel those obtained by a consensus of the 37
analysts each making a subjective assessment of the matches.
These results imply that the combination of (1) the structure of
the USE
(i.e., the linguistic hierarchical structure),
(2) the fuzzy set
mathematics, and
(3) a consensus approach
to assessing the fuzzy
sets themselves provided a reasonable representation of the subjec-
tive scoring of the same data by a large number of individuals.
A Quantitative Definition of Target Orthogonality
It
is often of interest to define how similar or dissimilar targets
are to each other. For example, free-response experiments like the
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
|
208
The Journal of Parapsychology
•
j
ganzfeld often use target packets, with the unselected targets in a
packet serving as decoys for judging. Assigning potential targets to
j
packets would be easier with some measure of target orthogonality.
Target definition for the purposes of this mode of analysis is ex-
actly the same as the one described
(i.e., a given target is defined by
its fuzzy subset of the USE, which has been coded to reflect the vis-
j
I
ual importance of each target element). The average number of ele-
ments, of the total of 13 1, that was assigned a nonzero value for the
j
targets in our pool of 200 was approximately 37, indicating that the
fuzzy
set representation of the target pool
is rich in visual infor-
mation. We used this information to determine the degree to which
the target set contains visually similar targets.
It
is beyond the scope of this paper
to describe the extensive
j
work in the literature seeking to find algorithmic techniques that
j
mimic human assessments of visual similarity. One recent article de-
scribes techniques similar to the one we used (Zick, Carlstein, & Bu-
I
descu, 1987).
.
j
We begin by defining the simlarity between target
i and target;
to be a normalized fuzzy set intersection between the two target sets:
^Wtmin{^A(T),^(7j)}j
Sij =
k
h
where the index k ranges over the entire USE. We have allowed for
the possibility of weighting the membership values with weights Wk
to examine various level/element contributions to the target similar-
itics.
For N targets, there are N(N~
1)12 unique values (19,900 for N
= 200) of Sij. The values
i and j that correspond to the largest value
of
represent the two targets
that “look” most similar. Suppose
j
another target m is chosen and Sm>i and Smj are computed. If both of
j
these values are larger than Sm n (for all n not equal to
i or j), then
target m is assessed to be most similar to the pair if The process of
grouping targets based on these similarities
is called cluster analysis.
Using
this
process, 200
targets were grouped
into
clusters,
such that the targets are similar within a cluster, and dissimilar be-
tween
clusters. Table
provides an overview of the
clusters
found from the total analysis of the 200 targets. Some of the names
appear to be quite similar, but, in fact, these sets are visually quite
distinctive. Figure 4 shows the graphic output of a single cluster in
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
'
Advances in Remote-Viewing Analysis
209
Table 2
Names of the 19 Clusters
No.
Name
No.
Name
Flat towns
Cities with prominent geometries
Waterfalls
Snowy mountains
Mountain towns
Valleys with rivers
Cities with prominent structure
Meandering rivers
Cities on water
Alpine scenes
Desert/water interfaces
Outposts in snowy mountains
Deserts
Islands
Dry ruins
Verdant ruins
Towns on water
Outposts on water
Agricultural scenes
detail. A much more complex—and
visually
difficult
to
under-
stand—graph
is generated for the
full
cluster
analysis and
is not
included here; this smaller subset, therefore, has been chosen to be
illustrative of the whole analysis. All targets in this particular sample
cluster are islands; the island in each photograph
is visible in its en-
tirety. Except for one outlier
(i.e., a hexagonal building covering an
island), the islands fall into two main groups
(i.e., with and without
1-Sy
Figure 4.
Cluster analysis of island targets.
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
210
The Journal of Parapsychology
manmade elements). The natural islands include three similar
mountain islands, two sandbars, and two flat verdant islands.
Using cluster analysis in conjunction with fuzzy set analysis pro-
vides for a quantitative definition of sets of targets that are similar
to each other within a cluster, but visually different across clusters.
Orthogonal clusters can be used to provide visual decoy targets for
traditional rank-order judging.
;
!
|
Recommendations and Conclusions
i
To apply the analysis in
its present form to a long RV series
is
quite labor intensive and, from the
results shown
in Figure
3,
is
most likely not justified since this fuzzy set technique approximates
human assessment. As we stated
in the introduction, however, we
are providing only a progress report of ongoing research. Because
of the decision concepts described in Dawes (1988) and the obvious
benefits of an automated evaluation system, the effort to improve
what was described in
this paper
is certainly justified. The proce-
dure can be used “as
is” to improve and quantify target orthogo-
nality.
Several future research areas are suggested to improve the tech-
niques described in this paper. The use of both inter- and intra-level
weighting factors needs to be examined systematically. In the analy-
j
sis described above,
all
levels and elements were accorded equal
j
weight. The ideal goal would be to determine the optimal weighted
mix of abstract versus concrete elements, as a means
to achieving
the following objectives:
. Refinement of the cluster analysis for targets, in an effort to
j
simulate, as closely as possible, what
is meant by “visual simi-
j
larities” between targets.
2. Refinement of the analysis of responses,
in an effort to
achieve even greater correlations between the fuzzy set figure
of merit analysis and various forms of ground truth.
!
Another area
that requires examination
in some
detail
is
the
_
|
USE and the hierarchical nature of its structure.
It is probable that
j
some elements are more appropriate than others; furthermore, they
|
might be more effectively structured in a semantic network as op-
posed
to a true hierarchy.
If a hierarchical structure
is
retained,
then some attention must be paid to the formulation of logical con-
sistency rules that govern element use. This would include numeri-
i
i
Approved For Release 2001/03/07
: CIA-RDP96-00791 R0002001 000034
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Advances in Remote-Viewing Analysis
211
cal relationships governing the membership values
(p/s) of higher-
order elements (e.g., port) vis-k-vis the combined value of their con-
stituent parts (e.g.,
city, river, boats, jetties, commercial).
One
inadequacy of the
system
is
that
it atomizes conceptual
“units.” For example, if the response element is red box, information
is lost in separating red from box. Current research in fuzzy set the-
ory indicates that fuzzy aggregates of fuzzy elements—“fuzzy sets of
fuzzy sets”—are mathematically complex but possible. Some effort
should be made to determine whether this technology could be im-
plemented as a means to capturing the information content of the
RV response with greater accuracy.
For the visual analysis, research into visual similarities between
pictures of natural scenes may serve as a potential refinement tool.
The aim here would be to enhance the visual orthogonality of rank-
order
analysis decoy
targets
as much
as
possible. Experiments
in
normal perception of similarities would
assist
in determining
whether scenes are perceived
as similar because of their low-level
geometries, concrete elements, or some combination of factors. The
ultimate aim would be to refine the target cluster analysis such that
it closely simulates ground truth representations of orthogonality.
Appendix A
Coding Forms for the Universal Set of Elements
The following coding forms illustrate the use of a universal set of elements
(USE) that matched our particular special targets, viewers, and requirements.
We constructed our USE by including a list of features present in photographs
from
the
National
Geographic
with elements
obtained from
the
remote-
viewing responses in earlier experiments.
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Figure Al. Coding form.
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 000034
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3
Advances in Remote-Viewing Analysis
213
Approved For Release 2001/03/07
: CIA-RDP96-00791R0002001 00003-3