A rational analysis of confirmation with deterministic hypotheses

A Rational Analysis of Confirmation with Deterministic Hypotheses Joseph L. Austerweil (Joseph.Austerweil@gmail.com) Thomas L. Griffiths (Tom Griffiths@berkeley.edu) Department of Psychology, University of California, Berkeley, Berkeley, CA 94720-1650 USA Abstract Whether scientists test their hypotheses as they ought to has in- terested both cognitive psychologists and philosophers of sci- ence. Classic analyses of hypothesis testing assume that peo- ple should pick the test with the largest probability of falsi- fying their current hypothesis, while experiments have shown that people tend to select tests consistent with that hypothesis. Using two different normative standards, we prove that seek- ing evidence predicted by your current hypothesis is optimal when the hypotheses in question are deterministic and other reasonable assumptions hold. We test this account with two experiments using a sequential prediction task, in which peo- ple guess the next number in a sequence. Experiment 1 shows that people’s predictions can be captured by a simple Bayesian model. Experiment 2 manipulates people’s beliefs about the probabilities of different hypotheses, and shows that they con- firm whichever hypothesis they are led to believe is most likely. Keywords: confirmation bias; rational analysis; hypothesis testing; Bayesian inference. How should a scientist seek evidence to help her find the hypothesis that explains a phenomenon? Does this differ from how people do seek evidence? Popper (1935/1990) ar- gued that scientists ought to follow the strategy of falsifica- tion, seeking evidence most likely to falsify their current the- ory. Interested in whether people adhere to this strategy, Wa- son (1960) investigated how people intuitively test their the- ories. In the classic 2-4-6 task, participants were asked to un- cover a relational rule after being told that one triplet, (2, 4, 6), conforms to the rule. The true rule, increasing numbers, sub- sumes most potential rules (e.g., two more than the previous number) with every triplet predicted by a potential rule also being valid under the increasing numbers rule. Thus, the true rule can only be discovered by testing numbers that are not predicted by your current best guess at the rule (negative test strategy or NTS). Rather than follow the NTS, participants choose to test triplets predicted by their current hypothesis (the positive test strategy or PTS) even though it is impossi- ble to find the true rule this way. For example, many partic- ipants in the 2-4-6 task followed the PTS by entertaining the hypothesis that each number is two more than the previous number and testing sequences consistent with this hypothe- sis, such as (1, 3, 5). The tendency to follow the PTS is just one instance of what has become known as the confirmation bias or the general human tendency to interpret and seek ev- idence fitting their current theory differently from evidence against it (Klayman & Ha, 1987). In this paper, we outline a set of environmental conditions under which the PTS is actually an optimal strategy. Previ- ous work has identified settings in which the PTS or NTS is more likely to yield falsification (e.g., Klayman & Ha, 1987). However, this normative analysis produces predictions that are quite different from human behavior. For example, peo- ple still use positive tests in situations where negative tests are more likely to yield falsification, such as those encountered in Wason’s (1960) experiment. We complement this analysis by showing that the PTS is more likely to yield falsification and optimally reduces uncertainty provided the world is in- herently deterministic (i.e., given the rule is true, there is only one possible next outcome). This suggests we might explain use of the PTS as the result of an assumption of determinism on the part of human learners, consistent with recent results showing that children assume that many causal relationships are deterministic (e.g., Schulz & Sommerville, 2006; Gel- man, Coley, & Gottfried, 1994). This emphasis on the struc- ture of the environment parallels similar strategies pursued in other rational analyses (e.g., Oaksford & Chater, 1994). The plan of the paper is as follows, first we introduce the task of predicting the next event in a sequence. Under the as- sumption that hypotheses are deterministic (given a sequence of events, a hypothesis predicts only one next event), we prove that the PTS is optimal in many situations. Next, we define a Bayesian model of sequence prediction for numeri- cal stimuli, and use a behavioral experiment to show that it captures human predictions. If people are seeking evidence optimally, then they should choose to verify the next num- ber predicted by the hypothesis they believe is most likely. In a second experiment, we demonstrate that changing a per- son’s beliefs about the probability of hypotheses affects their evidence-seeking strategy. We conclude by discussing how our results relate to previous work. Sequence prediction and hypothesis testing Given a sequence of events, how do we predict what will oc- cur next? For example, suppose you see a woman outside of an airport and then at the security checkpoint. How likely is it that she stays at the security checkpoint (she is a security guard) or walks to a gate (she is a passenger or crewmember)? Clearly, the probability of each possible next event depends on the probability of the hypotheses explaining the observed events and the probability of the next event under these hy- potheses. Since there is no means of predicting the next event with complete certainty, this is an inductive task. This problem can be expressed in terms of probability the- ory. Given a sequence of previous events or objects (~ x = (x 1 , . . . , x i−1 )) the probability of a next event (x i ) is P(x i |~ x) = ∑ P(x i |h,~ x)P(h|~ x) h where P(x i |h,~ x) is the probability of the next event under hy- pothesis h, and P(h|~ x) is the posterior probability of that hy- pothesis given the sequence ~ x. This posterior probability can