Individual Performance in Optimal Bayesian Inference

Individual Performance in Optimal Bayesian Inference Rakesh Patel (rakesh_patel_1@brown.edu) Department of Cognitive, Linguistic, and Psychological Sciences, Brown University Providence, RI 02912 USA Abstract Applications of Bayesian inference to human decision- making have met with mixed success, but new theoretical developments and experimental paradigms are helping to form a clearer picture of the role that inference plays in human cognition. We combine the latest ideas to provide evidence that at a computational level, the mind’s ability to make predictions may be grounded in Bayesian theory. Our results support the idea that the mind’s capacity for statistical reasoning is more sophisticated than previously hypothesized. Keywords: Bayesian inference; iterated learning; prediction; prior distribution. Introduction Imagine trying to predict your final grade in a math course, given only your grade on one homework assignment of many. Or consider being stuck in a traffic jam for 30 minutes, and deciding whether to wait until the traffic subsides or to exit the freeway and take slower surface streets. On a daily basis, we are bombarded by these types of situations in which we must make a decision on the basis of a very sparse set of data, often just one relevant observation. A powerful paradigm in cognitive science is the heuristics and biases approach (Gigerenzer, 1991, p. 84), which suggests that humans have difficulty making probability assessments. In particular, Tversky (1974) suggests that people cannot produce correct posterior probabilities on a simple Bayesian inference task with just 2 hypotheses (p. 1125). Instead, he claims, they produce estimates by transforming given data with simple linear or constant functions, a phenomenon known as the anchoring bias. He cites an experiment in which subjects were given either the number 10 or the number 65, and were asked if the number under- or over-estimated the percentage of African countries in the United Nations. Those who were given 10 as a starting number gave an average of 25 as the true answer, and those who started with the number 65 gave an average of 45 as the true answer. The anchoring effect explains this bias, Tversky claims, because people’s responses depended heavily upon initial values. Not content with Tversky’s classical view, some researchers now believe that people use given data and prior knowledge to make intelligent estimates in accordance with statistical theory. After all, if somebody asks you a question such as, “Do you think the population of Russia is greater or fewer than 180 million?” you assume that the person’s given estimate, 180 million, is not purely random and serves as a reasonable initial guess at the true answer. The estimate of 180 million is the only information you have, so why not make use of it by treating it as given data? When results such as these are analyzed from the perspective of Bayesian inference, many of the biases disappear. To perform a Bayesian inference on a set of data, one first hazards a guess on the distribution of the data. After observing some real-world data, the guessed distribution is then revised – its mean, density, and overall shape may differ considerably from the initial guess. When the real-world data consists of a single datapoint, known as a probe value, the median of the revised, or posterior, distribution given the data is the optimal prediction. Tenenbaum and Griffiths (2006) tested 350 college students’ ability to predict the outcome of everyday events (p. 767) by asking them questions such as such as, “Suppose that in 2000 B.C. a certain pharaoh has been ruling Egypt for 11 years. How long do you predict the total length of his reign to be?” After aggregating the results, they found that people’s responses closely matched the optimal predictions given the single datapoint contained in the question. The finding is remarkable because different phenomena have wildly different distributions and thus very dissimilar optimal prediction functions. Life spans are normally distributed, movie run times follow a power-law distribution, and durations of pharaoh reigns follow an Erlang distribution, for example (see Figure 1). Tenenbaum and Griffiths concluded that people implicitly store knowledge of the distributions of these everyday phenomena (p. 771), and use this knowledge to make optimal predictions. Mozer, Pashler, and Homaei (2008) presented the first theoretical challenge to these findings. They claimed that people’s responses followed real-world distributions only when aggregated, as in Tenenbaum and Griffiths’ analysis (p. 1134). As a demonstration of this effect, Mozer et al. recalled a country fair in which hundreds of people were asked to give the weight of an ox, and the average of their responses differed from the true value by just one pound (p. 1134). Their minimum-of-k-samples, or Min(k), algorithm produced the same results as in Tenenbaum and Griffiths’ experiment, even though it provided responses by simply choosing the minimum value from a very sparse (just 2 or 3) number of samples. Because the collection of everybody’s limited number of real-world samples must match the actual distribution of real-world data, their responses should also match the appropriate distributions when aggregated. When tested hundreds of times on Tenenbaum and Griffith’s questions, the Min(k) algorithm’s responses gave near- optimal predictions (p. 1145), even when assuming that each person might have access to just 2 samples. But because this algorithm assumes that prior knowledge is contained in just 2 or 3 samples, it should perform poorly