From: hrubin@pop.stat.purdue.edu (Herman Rubin)
Newsgroups: sci.med
Subject: "Scientific" evidence
Date: 23 Aug 92 13:12:57 GMT
Distribution: sci
Organization: Purdue University Statistics Department

In article <16283@ucdavis.ucdavis.edu> fzjaffe@othello.ucdavis.edu (Rory Jaffe) writes:
>In article <1992Aug22.212704.2936@news.Hawaii.Edu> lady@uhunix.uhcc.Hawaii.Edu (Lee Lady) writes:
>>In article <1992Aug21.142600.7122@spdcc.com> dyer@spdcc.com (Steve Dyer) writes:
>>>In article <1992Aug21.104014.3239@news.Hawaii.Edu> 
>>                              lady@uhunix.uhcc.Hawaii.Edu (Lee Lady) writes:
>>>> [It would be great is there was solid research proving that] 
>>>>therapies such as homeopathy and chiropracty don't work.

>>>That's the null hypothesis.  Research provides evidence (or not)
>>>that a therapy DOES work.  

>>The old "You can't prove a negative" platitude.  In fact, for practical
>>purposes one can.  If a reasonable study is done and the claimed benefit
>>does not show up at the 95% confidence level, as far as I personally am
>>concerned that's good enough to convince me that the therapy in question
>>is not worth my investing time and money in.  

>NO NO NO NO NO!  (Where is a statistician when you need one? :-) )
>The typical quoted confidence level is aimed at only one type of
>error--detected a difference when there really is none in the overall
>population.  Thus exceeding the confidence limit says that the
>investigator is more than 95% sure that the difference he/she found
>wasn't random.  Not exceeding 95% does not indicate that there is no
>true difference in the overall population--that is a different type of
>error("beta error").  Typically, a study should show no clinically
>important difference at >80%certainty to say that there is no effect.  I
>rarely see this statistic in negative studies, although it is important
>for proving equivalence.  One reason this statistic is rarely seen is
>that a study typically has to be pretty big to have a decent chance to
>exceed this 80% figure.

This is what happens when one uses the erroneous idea of "statistical
significance" as an indication of the worth of a treatment.  The level
of statisical significance has nothing to do with it.  Even the idea,
when looked at carefully without the brainswashing by classical statistics
courses, is ridiculous.

What a significance test purportedly tests is that there is ABSOLUTELY NO
effect.  Now I do not need any data to disbelieve this.  How much water one
drinks undoubtedly has an effect on cancer rate; now most people, including
me, believe that the effect is too small to worry about.  But a statistical
significance test on 10^100 people may very well find it.  Significant, but
unimportant, effects may be found in smaller samples as well.  If a drug
reduced blood pressure in 95% of the people by 0.01 mm (and this could be
measured), it statistical significance would very likely show up in relatively
small samples.

On the other hand, with a small sample, finding statistical significance for
even a moderately strong effect is not too likely.  This is why studies for
adverse effects are done at enormous doses, and the ASSUMPTION is made that
one can argue from these to normal doses.  The famous Canadian rat saccharine
study is of this type.  The problem with smaller samples or less extreme
situations is the difficulty of finding effects; the superabsorbent tampons
were tested on a reasonable number of women, and TSS did not occur in any
of those.

Now suppose someone suggests, and provides medical arguments for it, that
something will have long-term effects, say after 10-20 years.  One must use
means other than controlled experiments to decide this; the appropriate
controlled experiments for the effects of tobacco have not, and will not,
be performed.  SOME experiments which MAY be related have been.
-- 
Herman Rubin, Dept. of Statistics, Purdue Univ., West Lafayette IN47907-1399
Phone: (317)494-6054
hrubin@pop.stat.purdue.edu (Internet, bitnet)  
{purdue,pur-ee}!pop.stat!hrubin(UUCP)
Newsgroups: sci.med