From: wsadjw@rw7.urc.tue.nl (Jan Willem Nienhuys)
Newsgroups: sci.skeptic
Subject: Re: Statistical Evidence (was Re: Fwd: Astrology...)
Date: 8 Oct 92 16:02:10 GMT
Reply-To: wsadjw@urc.tue.nl
Organization: Eindhoven University of Technology, The Netherlands

In article <1992Oct7.220030.3484@Princeton.EDU> ydobyns@phoenix.Princeton.EDU (York H. Dobyns) writes:
[Finally something worthwhile discussing in this newsgroup]
>
>>Gauquelin's data originally comprised 2087 athletes. Expected number
>>of Mars athletes was 359, with a standard error of 17. Lots of physicists
>>don't think a deviation of less than 5 sigma merits serious investigation.
>
> (!!!)
>Maybe "lots" of physicists don't, but this physicist hasn't met many of
>them.  

I should qualify my statement.  When one is measuring something as 
a confirmation of a theoretical prediction (the workings of a self-designed
apparatus ceratinly qualifies as such) then physicists are much less 
demnanding.

But here we have the situation of `naturally produced data'  that have 
no theoretical prediction.  It resembles the situation of counting
neutrinos, detecting gamma rays from an otherwise unknown celestial
source; many examples of measurements of naturally occurring phenomena
come to mind.  In that case I still maintain that physicists are  very
wary of attaching theoretical importance to a 3 sigma peak in the noise.
(I just quote an astronomer I know.)

I *certainly* would not put up with someone who handed me a piece of
>apparatus and said "Oh, by the way, we tested the output and in terms of
>our measurement uncertainty it was only 4 sigma out of spec, so we figured it
>must be OK and didn't bother doing any more measurements." The overwhelming 
>majority of papers I've seen in physics are content to use at most 95% 
>error bars or the equivalent: that happens to be about 2 sigma for a 
>one-dimensional parameter measurement. Sometimes the conservative researcher 

How many of those papers report an utterly unknown and ununderstood 
new phenomenon on the strength of it exceeding the random noise level
by 2 sigma?

>reports a 99% confidence interval instead, that's about 2.6 sigma. I find 
>this statement of JWN's utterly outrageous.
>
>Significance-fetishists, eh? Well, let's apply a proper Bayesian
>approach to the numbers that Jan finds so unimpressive--statistical

[statistics lesson deleted.  Flame war about bayesian pseudo-science
 forestalled]

>Now I, personally, don't believe in astrology, and I have to admit that
>those numbers I've just calculated give me a sinking feeling in the
>pit of my stomach. So maybe the 2087/435/359 figures are also susceptible 
>to the accusations of data selection,

Well, IF this was a perfect 2087-fold Bernoulli experiment. But
at these numbers there are some problems.  The 17.2 percent refers
to long-time averages, and I don't know if the "binomial variance"
really is reliable in this case.  Secondly, the effect size is small,
(about 4%) and we have to ask ourselves how much certainty we have that
the researcher collecting the data cannot have made systematic errors
of that size.  He collected his data in batches of 20-100 over the course
of 20 years, and nobody knows exactly how he determined who was a good
athlete and who not.  But among the ones that he thought "not good enough"
there were significantly less than 17.2 Martians. 

This brings me back to the above discussion about 5 sigma/ 3 sigma. 
One can't predict how large one's systematic errors will be from
knowledge of the random errors.  But if the aggregate of all your
measurements with a given method seems off by only 3 sigma, I wish 
you much luck with tracking down what caused it: a genuine effect,
some kind of bias or a fluke or an erroneous estimate of the size of
sigma.  From what I know of observations of natural phenomena, 
variances are very often underestimated.

JWN