From: jones@REED.EDU (Albyn Jones)
Newsgroups: bit.listserv.edstat-l
Subject: CI's, conundrums, etc.
Date: 14 Oct 92 18:34:00 GMT
Reply-To: Albyn Jones <jones@REED.EDU>
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU

Murray A. Jorgensen [ maj@waikato.ac.nz ] writes:

>My case is that the amount of time and effort needed to
>communicate the precise meaning of the frequentist confidence
>machinery can lead to the impression that the technical details
>are the important part of statistics. They aren't.
>
>Statistics comprises several stages.
>
>1. A Question
>
>   [in a subject area such as Agronomy, Medicine, or Industry]
>
>2. Gathering data
>
>   [via experiment, survey, or observation]
>
>3.  The translation of the situation into mathematical terms.
>
>   [eg 3a  Deterministic formulation
>       3b  Frequentist formulation
>       3c  Bayesian formulation
>       3d  Fuzzy set formulation {God help us}
>       . . .]
>
>4.  Mathematical manipulation of the data within the formulation
>    of 3.
>    [4a,4b,4c,...]
>
>5.  Translation of the results of 4 back into the subject area
>    language.
>
>So when we speak in loud capital letters of what's RIGHT and what's
>WRONG we are speaking about concepts and derivations
>within a particular formalism, and not larger truths about
>this universe we live in. I'm not saying that one should not
>teach these concepts as clearly and accurately as you can. But
>let's not spend a lot of time on clever little conundrums
>designed to advance one particular formalism at the expense
>of another. And let's use the time that we save by looking at
>some important but only perfunctorily handled problems like
>how do we get from stage 2 to 3 [translating real problems
>into mathematical formalism] and from 4 to 5. It's my belief
>that many of the concepts we keep so nicely separated at
>stages 3 and 4 get mapped together at stage 5.
>
>Futhermore a chain of inference is only as strong as its weakest
>link, so to spend a lot of effort on making the formal part of the
>subject impeccable and to neglect the process of translating the
>real problems into the formalism because it is less clear-cut runs
>the risk of making statistics seem irrelevent.

Sounds pretty reasonable to me.  I don't spend a lot of time on "clever
conundrums", especially not with the goal of advancing one school of
inference over another, I'm mostly a data analyst rather than a
theoretician.  In introductory courses for non-math majors I hardly
ever mention Bayes methods at all, other than as an aside after
introducing Bayes' Rule. (Yes, I think students should do a little
probability along the way to doing statistics.) I do spend lots of
time on the process of translating substantive problems into statistical
models, data analysis, and translation back to substantive conclusions.
We seem to diverge mainly at the last step of your sequence, precisely
the point where you ask the question "what does it mean?"  Its easy to
talk nonsense at this point of the process if you don't understand the
underlying logic.

I reviewed a paper a few years ago written for a public policy journal
advocating a change in a federal rule.  The empirical question was one
of risk, how dangerous consumption of some possibly toxic quasi-food
item was.  The paper reviewed 10 studies, each of which reported a
significance test of the null hypothesis of "no risk", more precisely
no difference in risk between the experimental group and the control
group.  The argument presented in the paper came down to precisely the
claim that repeated failures to reject the null hypothesis constitute
strong evidence that the null hypothesis is true.  The estimated
effect sizes were non-trivial, and in the same direction, but all the
studies were done with small sample sizes.  Looking at the paper, I
came to the opposite conclusion, namely the paper actually presented
very strong evidence of a non-trivial risk.  The author had a little
trouble translating from the statistical formalism back to substantive
conclusions.

Agreed, this is not an example of damage done by
misunderstanding what a 95% confidence interval means.  It is an
example of the more general problem of failing to understand the
basic logic of statistical inference.  I use this example every year,
and it makes some students uncomfortable, because they come to the course
with the belief that statistics is a means of getting "the answer",
and any hint that they might have to think a bit is scary.  Most
students react very positively to such examples.  I don't see it as
a waste of precious time better spent talking about modeling, because
it is a part of talking about the translation back to the substantive
area.

albyn jones
jones@reed.edu
Newsgroups: bit.listserv.edstat-l