From: rdnelson@phoenix.Princeton.EDU (Roger D Nelson)
Subject: Re: Utts paper (part 1 of 3)
Originator: news@ernie.Princeton.EDU
Keywords: PSI experiments, meta-analysis
Organization: Princeton University
Date: Thu, 16 Apr 1992 23:29:32 GMT

In article <1992Apr15.194753.7228@rhrk.uni-kl.de> kring@efes.physik.uni-kl.de (Thomas Kettenring) writes:
>Part 3: Quotes and Opinions
>
>          Meta-Analysis of Psi Ganzfeld Experiments

Most of article is deleted.  I won't go into a lot of detail, but with
all due respect, I suggest anyone interested in meta-analysis,
parapsychology experiments, statistics, or anything else addressed in
Kettering's summary of Utts' article should read Utts' article.
It is worth noting that none of the statisticians who commented on the
article, including Diaconis (indicated by Kettering to be a "skeptic")
found the faults Kettering does.  Indeed, with the exceptions of Hyman
(skeptic) and Morris (parapsychologist) the comments
were quite positive -- which does not mean everyone decided
parapsychological effects are "real", only that Utts' presentation was
sound and useful.

>The difference between Honorton's and Hyman's method is
>essentially the same as between Radin and Nelson's method
>on one side and Jim Giles' and my opinions on the other side
>in the PEAR discussion we saw in sci.skeptic in February.
>(I wrote a lengthy article then, getting no answer.)

I have attached a copy of the answer to your article I wrote February
20.  My recollection is that I sent it to you via email, but it would
seem best that I now post it.  (Beg pardon if my memory is wrong and
this is a bandwidth-unfriendly repost.)

>So, this meta-analysis proves only that scientists tend to
>get the results they wish to get, and it is no reason to
>believe in psi.

This is the remark that most makes me feel people should read the Utts
article independently.  I think Kettering here is referring to that
article, but, of course, it is not a meta-analysis, nor is it an attempt
to get people to believe in psi.

>(Hyman in the above ganzfeld analysis did [flaw assessment] more intelligently:
>"Using cluster and factor analyses, the 12 binary flaw variables
>were combined into three new variables, which Hyman named
>General Security, Statistics and Controls".  That's not unlike my
>proposition in February.)

Right, very agreeable.  But Kettering is either ignoring or did not read
most of the discussions of the Hyman/Honorton perspectives.  Rosenthal,
for example, showed unambiguously that Hyman's work was itself deeply
flawed.  We all see the world through personal filters, of course, but I
suspect that most readers, perhaps especially true skeptics, would find
that Utts was very straight and clear.  Diaconis ended his comment this
way:  "Parapsychology is worth serious study.  First, there may be
something there ...  Second, if it is wrong, it offers a truly alarming
massive case study of how statistics can mislead and be misused.  Third
it offers marvelous combinatorial and inferential problems. ... Finally,
our budding statistics students are fascinated by its claims; the
present paper gives a responsible overview ...

***********

Here is the February 20 response to Kettering.  The numbers refer to
criteria used by Radin and Nelson to assess quality.  If anyone wants it, 
I can also provide the original article in which Kettering proposes his
preferred flaw analysis procedures.

}
}(15) tamper-resistant RNG employed
}Since I never heard of an RNG which was so bad that it did
}deliver more 0's than 1's or vice versa, points (1)-(5) and

It is trivial to make or find these.  Indeed it is difficult to make one
that is _not_ biased; that is what many of the calibration and control
aspects of these experiments address.

}(11) look to me like just preventing a tampered RNG, like (15)
}(but I don't know much about the subject of hardware RNGs).
}Tampering RNGs is probably a lot of work and requires
}specialized knowledge and real determination to cheat.
}Therefore it can only be done by experimenters, and malevolent
}ones at that.

I will be repeating myself now.
The point of the quality rating was to establish within the limits imposed
by the reports in the literature, some estimate for each experiment of
the care with which it was done.  Our procedures, the background, and
various concerns (e. g. what weights to use) are detailed in the
Foundations of Physics report.  
The purpose was in effect to assess objectively the conundrum posed by
critics to the effect that better controls result in smaller effects,
etc.  This has been an unsupported assertion, used to buttress the
conclusion that there is no effect, really.  The answer in this
meta-analysis (and in several others, see Utts, Statistical Science,
Nov. 91) is that effect size is not a function of quality.  The
"better control" argument, while logical in its own right, and 
appealing to the skeptical, is not sound.  
Indeed, one implication of our finding is that even the low
rated experiments were not typically defective in ways that predjudiced the
results.

}
}(12) data selection prevented by protocol or equipment
}General description of (6)-(10),(13), and (14).
}I don't know how to get this information.  General remarks by
}the authors or special measures which could have own criterion
}numbers?

This information, and all other information used for the quality
assessment is in the reports, as I have stated.  In the case of (12),
the reports indicated duplicate, independent records.  The PEAR
experiments have online computer files, and independent hardcopy on
continuous paper, and yet a third record of summary information for runs,
handwritten in bound logbooks.  Not all experiments have such depth of
protection from data selection, and we rated those with such
commensurately higher.

}(16) use of unselected subjects
}I'm not sure whether I understand this.

In the M-A, this criterion was used to reflect the criticism that
"special" subjects (e. g. self-proclaimed psychics) may be motivated to
cheat.  In most modern experiments this may be moot. 

}
}*Then* do this Monte Carlo simulation to give different weights
}to the criteria.
}This (or another similar way) would be an intelligent measure,
}using logic to reduce the number of parameters for the Monte
}Carlo simulation.
}This is better than just making a hot-pot of all criteria and
}let the MC simulation do the work, because one can now assess
}whether the SP parameter or the RNG parameter has an influence
}upon the effect size, and because now the MC simulation has a
}real chance of finding the right weights.
}

The point of the Dawes article I have previously cited is that the
particular choice of weighting scheme is not critical.  In effect,
looking for the "right weights" is the wrong question, essentially
because the criteria are arbitrary, though they may be sensible, and a
different set or a different set of weights will give the same "bottom
line" result.  Most people untrained in tests and measures have a
fallacious view of scores and ratings.  Crudely put, we tend to assume that
a score, perhaps especially one that looks objective, has inherent validity
and reliability.  Experience teaches that all such scores are _estimates_,
and drives home the reality of _variance_, which is far larger than the
naive observer expects (if she :-) expects any at all).