Newsgroups: bit.listserv.stat-l
Date:         Tue, 13 Oct 1992 09:14:00 EDT
From:         "Philip Gallagher,(919)966-1065" <UPHILG@UNC.BITNET>
Subject:      Re: Need help with PCA

    Immediate anguished unedited response:  "Thorough descriptive
    analyses are data snooping, and PCA isn't?!!!"
                                    ##########

    Now follows a slightly better reasoned response:
    ===============================================

    Andy Taylor feels that doing descriptive analyses of the data
    and then backing up the pictures some kind of inferential procedure
    are "data snooping".  My opinion is that PCA is data snooping.  As
    is any other inferential procedure done semi-blindly because the
    data will fit into the procedure without causing the computer to
    choke.

    1.  The essence of my recommendation was doing a thorough
        descriptive analysis, followed by the simplest inferential
        procedure possible - and that only because most editors have
        an uncritical need to see a p-value.  With a better designed
        experiment, better measurements, and (please God!) larger
        Ns (as will be done in the next round of experimentation),
        why then one will have a much better chance of doing a valid
        inferential test of the hypothesis.
    2.  I was quite serious above when I said that I consider PCA
        in the type of situation April describes as data snooping;
        is there anyone on the list who doesn't?
    3.  I long ago adopted an approach I learned in grad school (but
        I can't remember from whom, so I can't give credit/blame).
        Although we treat most research as "Hypothesis Testing", in
        truth most research is "Hypothesis Generation", in which one
        gathers a bit more information, refines the hypothesis/theory,
        does another expt., examines the results, refines the
        hypothesis, etc.  All this until, in principle, one has learned
        so much about the subject matter that one knows enough to be
        able to do a True Test Of The Hypothesis Experiment.
        I claim that, until this point, (also in principle) the
        so-called inferential tests we have done are better described
        (at least in our minds) as descriptive stats, leading not to
        final conclusions about the phenomenon but rather to the
        generation of a better hypothesis.
            (Note that in medical research (at least), once one
             knows enough about the phenomenon to be able to design
             a final True Test Of The Hypothesis Experiment, one
             usually knows so much about the phenomenon that further
             research cannot be done because it is unethical.  This
             puts me in the unhappy situation of feeling that I will
             almost never be privileged to undertake what my heart
             considers a true inferential procedure.)
        None of the foregoing should be taken to mean that I advocate
        sloppy and invalid use of statistical procedures because I
        claim that I am not really in a hypothesis _testing_
        situation.  One attempts to follow all the rules and get the
        best analyses one can, and then _interpret_ them as best one
        can, realizing the limitations that the research may have
        encumbered the data with.

    With April's data I am convinced that the best approach is to
    describe the data as well as possible (using, very carefully,
    even p-values from inferential procedures as _descriptive_
    statistics, not for hypothesis testing).  Then you throw in
    ONE p-value somewhere in the least noticeable place you can
    find in order to satisfy the (statistically naive?) editor,
    write the paper, and return to the lab to design a much
    improved data collection effort.  (Actually, I have discovered
    that editors cannot be trained to distinguish between p-values
    used as descriptive statistics and p-values used to test
    hypotheses, so I try to never let them creep into the papers
    as descriptive stats.  Gutless.)

    I know from whom I acquired the concept of p-values as descriptive
    statistics, but since he is not on this list, I am reluctant to
    name him in a forum where he has no opportunity to present his
    ideas in his own words.  Suffice it to say, the idea is not
    original with me, but the responsibility for adopting it is.

                             Phil Gallagher
                             uphilg@unc
Comments: Gated by NETNEWS@AUVM.AMERICAN.EDU