From: schabert@samish.Stanford.EDU (Vernon Schabert)
Subject: re: Psychology and statistics
Organization: DSO, Stanford University
Date: Tue, 25 Aug 92 18:58:00 GMT

Marc Green writes:

|> Statistics should be avoided if possible. Anybody who thinks that
|> statistics are "objective" knows very little about statistical theory
|> or its history.  Any use of statistics requires a host of assumptions,
|> not only about things like underlying distributions and metric
|> properties of the data, but about interpretation.

Yes and no.  Most Gaussian (normal-theory) statistical models make many assumptions about the distribution of one's data, but many nonparametric techniques exist that make fewer assumptions about data.  Also, statistical tests are often robust to violation of some of their assumptions.  If you've examined your data before modeling them with statistical procedures, you should know whether your data fit the assumptions of the model, and thus how much you should trust your modeled results.

Assumptions about interpretation are completely in the hands of the researcher.  Statistics should support a researcher's interpretation of data, but should not assume the role of interpretation themselves.  Unfortunately, many psychologists forget this.

|> The use of statistics has other problems too. It usually means
|> averaging subjects together. It is easy to get "results" which do not
|> reflect any individual subject and it is easy to miss secondary
|> effects. 

A statistic is an estimate of a parameter from a population.  Why someone would use statistics to reflect the qualities of a particular individual is not completely clear.  Statistical models allow us to generalize about a population, provided the sample is truly reflective of the population.  If one wants to know about a particular individual, case studies would be more appropriate.

If secondary effects are detectable with your sample, then their variance is likely to obscure your data to the point that you can't detect your primary effect, unless you have also included them in your model.  A good example of modeling "secondary" effects is analysis of covariance, and other models exist for other situations.

|> Statistics are used as a means of control. But they are not a
|> substitute for experimental control, and too often, this is exactly
|> how statistics are used. The best experiments use a few subjects with
|> tight experimental control.

Again, yes and no.  Statistics are used as an arbitrary yet objective method by which to support arguments made by the researcher.  Statistical modeling itself has nothing to do with control; both rigorously controlled experimental studies and loosely bounded correlational data can be modeled by statistical methods.  Control, like interpretation, is completely in the hands of the researcher.  Lack of control means that relationships should be interpreted as relationships, and nothing more.  When other pote

ntial contributing factors have been controlled for in an experiment, then relationships may be more appropriately interpreted causally.

|> >In parting, I would also like to
|> >comment about post hoc "fishing" expeditions that presumably occur
|> >frequently in Social Psychology.  Post hoc explanations are not necessarily a
|> >bad thing.  No researcher is clever enough to think of all possible effects
|> >that might occur in particular study.
|> 
|> If using statistics correctly, fishing expeditions have a high cost.
|> Suppose you have a .05 level for significance in testing a specific
|> hypothesis. If you decide to test a second hypothesis with the same
|> data, this gives you two chances of getting significant result due to
|> chance. As a result, you should cut the statistical level of
|> significance to 0.25 (0.5/2). The more fishing you do, the more likely
|> that the result is due to chance, so significance level should be
|> adjusted downward. Of course, no one does this. That's the other
|> problem with using statistics - very few people really know how to do
|> statistics correctly. 

Obviously, the best plan is to know before looking at your data exactly which relationships to explore.  Then, after confirming via exploratory data analysis that a given statistical model is appropriate, you can test only those specific questions you have, pre-planning the appropriate contrasts and dividing up one's experiment-wise error rate appropriately.  This does not mean, however, that post-hoc tests should be avoided at all cost.  You've gone to all that trouble to collect that data; why not explor

e it further to examine other potential relationships?  Many post-hoc tests (such as Scheffe's contrasts) already adjust error rates to allow for all possible comparisons, so you needn't get into an alpha-level snit.

It is common and accepted practice (on the part of both psychologists and real statisticians) to disassociate error levels in post-hoc testing from those in a priori testing.  Post-hoc testing will never (or should never) be used to confirm an a priori hypothesis; it merely suggests reasons why your statistical model may not have been able to support your hypothesis in the first place.  Fear about accumulating error rates can easily get out of hand, with researchers afraid to run multiple studies because o

ne of their results might come about by chance.  Replication is actually the best way to discount the chance argument; getting the same results with different data sets increases one's ability to argue that an effect or relationship is genuine, and not a product of statistical wizardry or freaks of probability.

Vernon Schabert
schabert@psych.stanford.edu

Newsgroups: sci.psychology