Significance Tests and the Duplicity of Binary Decisions

Robert FolgerA. B. Freeman School of Business, Tulane UniversityPresents a logical justification for the following statements and discusses their implications: It isduplicitous (misleading) to use significance tests for making binary (either/or) decisions regardingthe validity of a theory; the binary choice between calling results significant or not significant shouldnot govern the confidence placed i n a theory, because such confidence cannot be gained in the either/or fashion characterizing deductive certainty. The implications include grounds for describing waysthat effect size estimates become useful in makingjudgments about the value of theories.Charles Sanders Peirce, who some say was America's greatestphilosopher of science, wrote that "science can only rest on ex-perience"; he then added that "experience can never result inabsolute certainty, exactitude, or universality" (Buchler, 1955,p. 47). Hypothesis-testing procedures make explicit science'slack of certainty. In particular, inferential statistics provide esti-mates of whether a finding reflects merely random variation, anoutcome owing to "chance" rather than to causal mechanismshypothesized by theories. I contend that such procedures canbe used in a manner that would deserve the label of duplicity(i.e., an attempt to dupe the unsuspecting) if it were actuallyintentional. Inferential statistics can be especially misleadingwhen carelessly applied logic suggests an unwarranted degreeof deductive certainty.Chow's (1988, Tabl e 2) syllogistic description of hypothesistesting (see my Table 1) illustrates how unwarranted conclu-sions can be reached by seemingly logical steps. Syllogisms areeither valid or invalid, hence "logical validity is an all-or-noneproperty of an argument" (Chow, 1988, p. 109). Chow therebymirrors common practice by emphasizing the binary choice be-tween rejecting or not rejecting a theory. Researchers followingcommon practice can dupe themselves and others, however, be-cause neither a theory's validity nor its invalidity is a matter ofdeductiv e certainty.Chow's (1988) idea that "the role of a statistical analysis. . .is to supply the investigator with the minor premise for the syl-logistic argument" (p. 108) also raises issues about the use ofeffect sizes:All that is required of a statistical analysis is a binary decision.. . .because the validity of the syllogistic argument requires only thatinformation. Even if a quantitatively more informative index isavailable (e.g., effect size, the amount of variance accounted for, orthe power of the test), it will still be used in a binary manner.. . .Nothing is gained by using an effect-size estimate.. . .(p. 108)Chow claimed that significance tests are superior to effect sizesI thank Bill Dunlap and Irving LaValle for comments.Correspondence concerning this article should be addressed to Rob-ert Folger, A. B. Freeman School of Business, Tulane University, NewOrleans, Louisiana 70118.on the basis of the logic of syllogistic argument. I interpret thesame syllogism differently, making the role of effect sizes oncemore an open issue.No Proof of Validity: Problems inAffirming ConsequencesTh e heading of th third colum n i Tabl 1 (Affirming conse-quent) is also th e nam of a logical fallacy, which should imme-diately raise suspicions about imposing either/or (binary) deci-sions when judging theories. The fallacy of affirming the conse-quent involves reasoning backward from the validity ofconclusions to th e presumed validity of their premises. Suppossomeone claims that "if we follow the principles of supply-sideeconomics, th economy will prosper" becaus "th election ofa supply-side President caused economic prosperity." Amongother interpretive problems (e.g., what does it mean to imple-ment supply-side policies; what measure reveals the degree towhich the President had in fact done so; and how should eco-nomic prosperity b e measured?), ther is a clea r dange of spuri-ous correlation: How can one be sure that prosperity was notcaused by something else? It is also easy to see the danger ofaffirmin g the consequent from example s of fallaciou reasoninsuch a s th following: "If Hitler i still alive, sun hot.""The sun is hot." "Therefore Hitler is still alive."Chow argued for all-or-nothing, dichototnous choices, evenwhil e using th wor d probably twic i n thir colum f Ta-ble 1, yet the reference to probability rather than to certaintyis demanded by the fallacious nature of consequence-affirmingarguments: Because it is logically false to conclude with deduc-tive certainty that true premises are entailed by true conse-quences, then at best any evidence for the consequences' valid-ity provides only arguments consistent with the validity of thepremises. It is more accurate to say that experimental resultsare consistent with a hypothesis than to say that they con-firm it.