From: ydobyns@phoenix.Princeton.EDU (York H. Dobyns)
Subject: Statistical Evidence (was Re: Fwd: Astrology...)
Originator: news@nimaster
Organization: Princeton University
Date: Wed, 7 Oct 1992 22:00:30 GMT

In article <5874@tuegate.tue.nl> wsadjw@urc.tue.nl writes:
[...accusations by JWN of lies in the original posting, and of bias
on the part of Ertel and the Gauquelins, deleted; my concern is
only with some numeric, statistical assertions:]

>Gauquelin's data originally comprised 2087 athletes. Expected number
>of Mars athletes was 359, with a standard error of 17. Lots of physicists
>don't think a deviation of less than 5 sigma merits serious investigation.

 (!!!)
Maybe "lots" of physicists don't, but this physicist hasn't met many of
them.  I *certainly* would not put up with someone who handed me a piece of
apparatus and said "Oh, by the way, we tested the output and in terms of
our measurement uncertainty it was only 4 sigma out of spec, so we figured it
must be OK and didn't bother doing any more measurements." The overwhelming 
majority of papers I've seen in physics are content to use at most 95% 
error bars or the equivalent: that happens to be about 2 sigma for a 
one-dimensional parameter measurement. Sometimes the conservative researcher 
reports a 99% confidence interval instead, that's about 2.6 sigma. I find 
this statement of JWN's utterly outrageous.

>That would mean that an experimental finding of over 446 Mars athletes
>starts to be interesting.  Initially Gauqulin found 452, but after
>recomputation (Correlation 4, 1983) there were only 435.  The whole
>effect is therefore so much at the border of significance that it hardly
>woth so much effort.

[...JWN concludes with snide remarks directed at Ertel's use of 1.65
sigma--the standard 95% one-tailed confidence level--as a significance
criterion, and with a sneer against "significance-fetishists."]

Significance-fetishists, eh? Well, let's apply a proper Bayesian
approach to the numbers that Jan finds so unimpressive--statistical
significance is a concept that doesn't even appear in that
formalism. The data in JWN's posting reproduced above report a
set of 2087 Bernoulli trials in which the theoretical expectation
is that 359 should fit a given criterion. In fact 435 cases fitting the
criterion are observed. What can we say about p, the probability that
one of these random trials fits the criterion (i.e., the probability
that one of these outstanding athletes has Mars in a "meaningful"
position by astrological standards)?

The null hypothesis ("There is no Mars effect") is that p=0.1720.
Call this H0.

A completely uninformed alternative ("There might be a Mars effect,
but there is no information we can use to predict its magnitude in
advance of the experiment") is simply 0<p<1.  Call this H1.

A somewhat better informed alternative hypothesis might be:
"Well, the Mars effect shouldn't be overwhelming, because if 
astrological effects were huge and unvarying there wouldn't be any
debate about the validity of astrology. And a result *less* than
the chance level hardly counts as evidence for a 'Mars effect'--it
refutes astrological predictions as surely as a null result would.
So let's say the plausible range of values for p is 0.172 < p < 0.344,
from the chance level up to twice the chance level." Call this
informed alternative H2.

The next step is to calculate the relative likelihoods of these
hypotheses given the observation of 435 "hits" in 2087 attempts.
The absolute values aren't interesting, because the useful information
is in how much the evidence favors one hypothesis over another--this
is called an odds ratio, odds adjustment, or Bayes factor depending
on who you consult. My figures are probably a few percent off, because
I'm using a normal approximation to the likelihood rather than the
actual beta-function. (They're not far off, though, because the
normal is going to be a fairly good approximation for totals this
large.)

The comparison between H0 and H1 produces an odds ratio of 96 in favor
of H1. That is, however confident I was that there is no Mars effect,
I should be almost 100 times less confident after reading the figures
Jan has presented.

The comparison between H0 and the more plausible alternate H2 makes the
odds ratio 560 in favor of the alternate. Same meaning as above, but
with a bigger number to plug in.

Now I, personally, don't believe in astrology, and I have to admit that
those numbers I've just calculated give me a sinking feeling in the
pit of my stomach. So maybe the 2087/435/359 figures are also susceptible 
to the accusations of data selection, etc., that JWN mentions with regard 
to some other figures. But I certainly can't salvage my peace of mind
by flying in the face of statistical inference and claiming, along with
JWN, that the observation *as presented by him* doesn't constitute
evidence for an effect.

York H. Dobyns		ydobyns@phoenix.princeton.edu