Sensible reasoning in two tasks: Rule discovery and hypothesis evaluation

The hypothesis testing skills of undergraduates were measured in two tasks: the 2-4-6 rule discovery task in which students generate and assess hypotheses, and a hypothesis evaluation task, which requires only the assessment of hypotheses. The results of Experiments i and 2 show that the students consistently employed a disconfirmation strategy when assessing hypotheses, but employed a counterfactual inference strategy when they also were required to generate the hypotheses. The results of Experiment 3 suggest that the selection of the hypothesis testing strategy reflected a balance between the logical requirements of the task and the desirability of possible outcomes. Taken together, the findings support a more consistent picture of human rationality across tasks, and suggest alternatives to accounts of confirmation bias.

Related papers:

title: Psychology of the scientist: An analysis of problem-solving bias
abstract: The problem-solving skills of 30 Ph.D. scientists were compared to those of 15 conservative Protestant ministers. Of particular interest was the frequency with which these groups generated confirmatory (rather than disconfirmatory)experiments to test their hypotheses. Experimental results showed that—contrary to a popular assumption—the reasoning skills of the scientists were not significantly different from those of nonscientists. In this study, the ministers showed a longer latency to speculation and generated more experiments per hypothesis than did the scientists. These findings are discussed in terms of the psychology of the scientist.

title: Sensible reasoning: A hypothesis about hypotheses.
abstract: TsCHmcI, JUDrTH E. Sensible Reasoning: A Hypothesis about Hypotheses. CHILD DEVELOPMENT, 1980, 51, 1-10. This study investigated the asserted differences in reasoning between adults and second, fourth, and sixth graders in a manipulation-of-variables task using common everyday situations. It is proposed that hypothesis-testing skills used in this task do not necessarily develop from an understanding of the logic of classes. These skills may develop through experience in manipulating elements in those naturally occurring situations where it is desirable to eliminate negative outcomes or reproduce positive outcomes. Subjects were given 8 multivariate stories with different outcomes, either good or bad, and asked to choose a test to determine which element caused the outcome. Subjects were expected to choose a test systematically on the basis of sensible considerations of the outcome value of any situation, not on purely logical grounds. For all age groups the general pattern of responses was the same. A logical disconfirming test (vary only the hypothesized variable) was preferred in bad-outcome situations, and a logically inappropriate confirming test (repeat the hypothesized variable in a different context) was preferred in good-outcome stories. The main developmental trend was a shift in choice of strategy to eliminate the hypothesized variable in bad-outcome stories. There was no relation between children's performance on class-inclusion problems and their story-problem solutions. These results are discussed in terms of "sensible" reasoning and problem-solving skills.

title: Strategies of Rule Discovery in an Inference Task
abstract: It has long been known that subjects in certain inference tasks will seek evidence which can confirm their present hypotheses, even in situations where disconfirmatory evidence could be more informative. We sought to alter this tendency in a series of experiments which employed a rule discovery task, the 2-4-6 problem first described by Wason. The first experiment instructionally modified subjects confirmatory tendencies. While a disconfirmatory strategy was easily induced, it did not lead to greater efficiency in discovering the rule. The second experiment introduced subjects to the possibility of disconfirmation only after they had developed a strongly held hypothesis through the use of confirmatory evidence. This manipulation also failed to alter the efficiency of rule discovery. In the third experiment, subjects were taught to use multiple hypotheses at each step, in the manner of Platt's “Strong Inference”. This operation actually worsened performance. Finally, in the fourth experiment, the structure of the problem was altered slightly by asking subjects to seek two interrelated rules. A dramatic increase in performance resulted, perhaps because information which in previous tasks was seen as merely erroneous could now be related to an alternative rule. The four studies have broad implications for the psychological study of inference processes in general, and for the study of scientific inference in particular.

title: On the Failure to Eliminate Hypotheses in a Conceptual Task
abstract: This investigation examines the extent to which intelligent young adults seek (i) confirming evidence alone (enumerative induction) or (ii) confirming and discontinuing evidence (eliminative induction), in order to draw conclusions in a simple conceptual task. The experiment is designed so that use of confirming evidence alone will almost certainly lead to erroneous conclusions because (i) the correct concept is entailed by many more obvious ones, and (ii) the universe of possible instances (numbers) is infinite. Six out of 29 subjects reached the correct conclusion without previous incorrect ones, 13 reached one incorrect conclusion, nine reached two or more incorrect conclusions, and one reached no conclusion. The results showed that those subjects, who reached two or more incorrect conclusions, were unable, or unwilling to test their hypotheses. The implications are discussed in relation to scientific thinking.

title: Disconfirmation and Dual Hypotheses on a more Difficult version of Wason's 2–4–6 Task
abstract: This paper reports two experiments in which subjects worked to solve a more difficult version of Wason's 2–4–6 task: instead of the usual “numbers must ascend in order of magnitude” rule, a more general rule, “the three numbers must be different”, was used. The first experiment established that instructing subjects to disconfirm on the “three different numbers” did not significantly improve their performance, as compared with confirmatory and control groups. Disconfirmatory subjects did try to propose more triples at variance with their hypotheses but were unable to obtain the necessary disconfirmatory information. To help subjects represent the task in a way that facilitated disconfirmation, the second experiment utilized a procedure in which subjects were told that they were looking for two rules, Dax and Med–the Dax rule corresponding to “three different numbers” and the Med rule to its complement, i.e. two or more numbers the same. Of subjects in the Dax–Med condition, 88% solve the rule, as opposed to 21% of subjects in a control condition. Dax–Med subjects tended to search for positive instances of the Med rule, which, in turn, forced them to test the limits of the Dax rule. It was concluded that the Dax–Med manipulation did facilitate a different mental representation of the task than the normal procedure.

title: Confirmation Bias in a Simulated Research Environment: An Experimental Study of Scientific Inference
abstract: Numerous authors (e.g., Popper, 1959) argue that scientists should try to falsify rather than confirm theories. However, recent empirical work (Wason and Johnson-Laird, 1972) suggests the existence of a confirmation bias, at least on abstract problems. Using a more realistic, computer controlled environment modeled after a real research setting, subjects in this study first formulated hypotheses about the laws governing events occurring in the environment. They then chose between pairs of environments in which they could: (I) make observations which would probably confirm these hypotheses, or (2) test alternative hypotheses. Strong evidence for a confirmation bias involving failure to choose environments allowing tests of alternative hypotheses was found. However, when subjects did obtain explicit falsifying information, they used this information to reject incorrect hypotheses.

title: Consequences of Confirmation and Disconfirmation in a Simulated Research Environment
abstract: Advanced undergraduate science majors attempted for approximately 10h each to discover the laws governing a dynamic system. The system included 27 fixed objects, some of which influenced the direction of a moving particle. At a given time, any one screen of a nine-screen matrix could be observed on a plasma display screen. Confirmatory strategies were the rule, even though half the subjects had been carefully instructed in strong inference. Falsification was counterproductive for some subjects. It seems that a firm base of inductive generalizations, supported by confirmatory research, is a prerequisite to useful implementation of a falsification strategy.

title: How the possibility of error affects falsification on a task that models scientific problem solving
abstract: This paper reports an experiment in which groups of four subjects attempt to solve two related rules when they know that 0–20 per cent of the feedback they receive may be in error. Groups were randomly assigned to one of three strategy conditions: confirmatory, in which subjects were urged to propose experiments consistent with their hypotheses; disconfirmatory, in which subjects were encouraged to propose experiments at variance with their hypotheses; and a control strategy, in which subjects were merely urged to test their guesses by proposing more experiments. 
 
 
 
In a previous study using the same rules and strategy conditions, but no error, Gorman et al. (1984) found that disconfirmatory groups performed significantly better than groups in other conditions. In the study described here, this performance difference did not arise. About 60 per cent of the time groups immunized their hypotheses against disconfirmation by classifying disconfirmatory results as error. About 40 per cent of the time groups correctly recognized that there was no error, but spent so much of their time replicating experiments to make sure that there was no error that they failed adequately to test their hypotheses. The few successful groups used a strategy that combined disconfirmation with replication. Detailed examples of how groups handled error are discussed, as are the implications of these results for future research.

title: A Philosophical and Empirical Analysis of Subjects’ Modes of Inquiry in Wason's 2–4–6 Task
abstract: Investigations of hypothesis-testing behaviour typically conclude that subjects’ methods are characterized by “confirmation bias.” However, these studies (a) relied exclusively upon Karl Popper's philosophy of science, (b) only described subjects’ strategies via a falsificationist terminology, and (c) failed to determine subjects’ precise intentions in the task situations. Alternative philosophies of science can be used to describe hitherto ignored aspects of rule discovery in Wason's (1960) 2–4–6 task as well as reinterpret findings from previous research. Using Wason's task, Experiment 1 adopted J. S. Mill's inductive methodology in order to assess better the extent of elimination in experimentation. While it was found that subjects did eliminate a large proportion of their hypotheses rather than merely confirm them, Mill's eliminative methods applied to this task failed to account for the entirety of subjects’ behaviours. Experiment 2 examined subjects’ methods in more depth by asking subjects during the task either to: (1) describe their procedures, (2) select from a list of methodology terms those that best characterized their methods, and (3) state on each learning trial their preferred hypotheses and their subjective assessments that those hypotheses were correct. Although subjects were generally unable to describe their methods of inquiry, they were able to typify them via the methodology terms as if their methods were tacitly held skills. Contrary to presumptions of previous research, analyses indicated that subjects were not testing particular hypotheses on every trial but were often examining possible instances of the rule “at random” or “different from ones examined” to that trial. It is also suggested that subjects’ decisions to announce rules and their thematic lines of inquiry be given further consideration by researchers.

title: Rationality and Intelligence
abstract: Preface 1.  Rational choices and plans 3. A theory of thinking 4. The scheme fleshed out: a decision-theoretic analysis of thinking 5. Conditions of effective thinking 6. Effects of rational thinking on the individual and society 7. The teaching of rational thinking References Index.

title: Confirmation, Disconfirmation, and Informa-tion in Hypothesis Testing
abstract: Strategies for hypothesis testing in scientific investigation and everyday reasoning have interested both psychologists and philosophers. A number of these scholars stress the importance of disconfir. marion in reasoning and suggest that people are instead prone to a general deleterious "confirmation bias" In particula~ it is suggested that people tend to test those cases that have the best chance of verifying current beliefs rather than those that have the best chance of falsifying them. We show, howeve~ that many phenomena labeled "confirmation bias" are better understood in terms of a general positive test strate~. With this strategy, there is a tendency to test cases that are expected (or known) to have the property of interest rather than those expected (or known) to lack that property. This strategy is not equivalent to confirmation bias in the first sense; we show that the positive test strategy can be a very good heuristic for determining the truth or falsity of a hypothesis under realistic conditions~ It can, howeve~ lead to systematic errors or inefficiencies. The appropriateness of human hypotheses-testing strategies and prescriptions about optimal strategies must he understood in terms of the interaction between the strategy and the task at hand.

title: Rationality and Disconfirmation
abstract: In a recent article in Social Studies of Science,' Michael Mahoney argues that scientists are not the paragons of intellectual and moral virtue that some have claimed them to be, but share to a considerable extent the vices of more ordinary folk. Readers of this journal will be familiar with the general range of considerations which Professor Mahoney adduces, so I shall not rehash them here. I wish, rather, to draw attention to some features of that section of his article which deals with the rationality of scientists. His discussion of this topic is flawed, it seems to me, by inaccuracies in the use of logical terminology, by a too-simple philosophical account of rationality, and by his appeal to an experiment of dubious design. The inaccuracy of terminology, which is even more apparent in NMahoney's earlier book, Scientist as Subject,2 is essentially a surface problem, but it needs to be set straight. For example, Mahoney asserts that 'Modus ponens and modus tollens are the only valid forms of inference, and the latter appears to be the only real option for the practising scientist.'3 Strictly speaking, modusponens and modus tollens are only two of infinitely many valid forms of deductive inference, and those other valid forms are not reducible to modus ponens and modus tollens even if we confine ourselves to the relatively simple forms of inference found in propositional logic. Far from its being the case that rnodus tollens is the 'only real option for the practising scientist,' a scientist (or anyone else) who confined himself to modus tollens would scarcely be able to engage in any reasoning  Mahoney's confusion seems to have its root in a mistaken identification of the truth-functional connective '-_.,commonly used to represent the weakest construal of the English 'If...then _', with the much stronger 'implies' or 'entails'. Thus, in Scientist as Subject, he says:

title: Surface structure and presupposition
abstract: Frequency of perceptual errors as a function of differential presupposition in descriptive sentences was measured. In Experiment I, subjects heard a cleft or pseudocleft sentence prior to tachistoscopic presentation of a picture in which the presupposed proposition or the focused proposition was misrepresented. The task was to decide if the sentence correctly described the picture. Results indicated that more errors occurred when the misrepresentation involved the presupposition than when it involved the focused proposition. Experiment II was conducted to determine if the frequency of errors was a function of the type of surface structure employed to mark the presupposition. Results indicated that several different surface structure features of English, employed in combination, increase the presuppositional strength of the sentence.

title: Science and the limits of logic: A response to the Mahoney—DeMonbreun paper
abstract: Perhaps the reader of the Mahoney-DeMonbreum paper is as discomfitted as I was when I first read it. It is an intriguing and somewhat chastening effort. At the same time, the perspicacious reader may have the gnawing doubt that he's been "had , " that there is an elusive feature about the experimental paradigm that is subtly problematic and misleading. I suspect that this feeling of uneasiness stems from the conviction that scientists are surely better versed in scientific inquiry than fundamentalist ministers, and that an experiment that (apparently) falsifies this conviction must somehow be unrepresentative of the true state of affairs. Nevertheless, data are data, and since psychologists are often very impressed with empirical results, such doubts may never surface and assert themselves. But stay--the doubts are justified, even though we can learn much from the experiment. My remarks will first concentrate on analysis of the experiment itself and then shift to an examination of how well it embodies scientific inquiry generally.

title: Psychology of the Scientist: An Evaluative Review
abstract: Although the social processes in scientific inquiry have received extensive analysis, psychologists have devoted relatively little attention to the thoughts, feelings, and actions of the individual scientist. This neglect has resulted in an unfortunate failure to evaluate long held assumptions about scientist behaviour. This article reviews sociological, archival, and recent experimental evidence bearing on the psychology of the scientist. These data suggest that the correspondence between scientist behaviour and accepted scientific 'ideals' may be far less than has been presumed. After briefly reappraising those ideals, it is argued that psychological research and particularly psychological theorizing are critical to an adequate understanding and refinement of human factors in science.

title: Rationality and Authority: On the Confusion of Justification and Permission
abstract: During the past several years I have elected to offer several comments on various psychological dimensions in science. These have ranged from technical pieces to intentionally value-laden openings. It is my enduring belief that contemporary science is seriously tethered by its own rigidity and more centrally by its reluctance to examine some of its inescapably insecure human foundations. The intensity and prevalence of emotionality in science needs little additional documentation, and I am not as convinced as some of my colleagues that this emotionality is a uniformly constructive element. When I read a contemporary debate or critique in science I am often struck by the contrast of its tenor to that apparent in the dialogues of Plato or Hume. While hardly immune to ad hominem overtones, these thinkers seem to have been much less preoccupied with demonstrating the mental incompetence of their conversants than some contemporary writers. It is, of course, difficult to separate message from source and as a psychologist 1 should perhaps be more understanding of such conflations. Thus, despite my genuine respect for the role of dialectical exchange in the growth of science, I find myself reluctant to respond to or otherwise participate in a dialogue in which I am portrayed as a butcher who has been 'hacking' away at scientific complexities 'with neo-Popperian meat-cleavers." But having said that I would not respond to this, I have obviously done so and have thereby consumed more space than I care to devote to surplus meanings and personal innuendos. It is my impression that Professor Hardin has elected to focus his technical remarks on the relationship of rationality and disconfirmation. Many of his arguments are directed at specific points distilled from a variety of scattered quotes. I shall respond to each of these briefly before offering a more holistic response to his more basic concerns. First, I am indeed guilty of overstatement and technical inaccuracy in my assertion that 'mnodus ponens and moduis ftolens are the only valid forms of inference.'2 This statement came out of an implicit context of convention (namely, the reign of Aristotelian thought in western philosophy and the unchallenged popularity of these two forms of inference). Within that context, I believe the statement remains accurate. If we shift to the abductive inference of the Stoics or the non-polar logic of some eastern perspectives, the number of inference forms considered valid by their generative systems becomes considerably larger. Hardin's second point on the truth-functional connective seems to be a reiteration of the difference between inferential necessity and sufficiency in material implication. I am more than willing to recognize this difference, but I question this invocation that we mix formal and informal meanings of the terms 'valid' and

title: Can human irrationality be experimentally demonstrated
abstract: The object of this paper is to show why recent research in the psychology of deductive and probabilistic reasoning does not have "bleak implications for human rationality," as has sometimes been supposed. The presence of fallacies in reasoning is evaluated by referring to normative criteria which ultimately derive their own credentials from a systematisation of the intuitions that agree with them. These normative criteria cannot be taken, as some have suggested, to constitute a part of natural science, nor can they be established by metamathematical proof. Since a theory of competence has to predict the very same intuitions, it must ascribe rationality to ordinary people.Accordingly, psychological research on this topic falls into four categories. In the first, experimenters investigate conditions under which their subjects suffer from genuine cognitive illusions. The search for explanations of such performance errors may then generate hypotheses about the ways in which the relevant information-processing mechanisms operate. In the second category, experimenters investigate circumstances in which their subjects exhibit mathematical or scientific ignorance: these are tests of the subjects' intelligence or education. In the third and fourth categories, experimenters impute a fallacy where none exists, either because they are applying the relevant normative criteria in an inappropriate way or because the normative criteria being applied are not the appropriate ones.