Approximating Bayesian inference with a sparse distributed memory system

Approximating Bayesian inference with a sparse distributed memory system Joshua T. Abbott (joshua.abbott@berkeley.edu) Jessica B. Hamrick (jhamrick@berkeley.edu) Thomas L. Griffiths (tom griffiths@berkeley.edu) Department of Psychology, University of California, Berkeley, CA 94720 USA Abstract we take on this challenge by showing that an associative memory using sparse distributed representations can be used to approximate Bayesian inference, producing behavior consistent with a structured statistical model while using distributed representations of the kind normally associated with artificial neural networks. Probabilistic models of cognition have enjoyed recent success in explaining how people make inductive inferences. Yet, the difficult computations over structured representations that are often required by these models seem incompatible with the continuous and distributed nature of human minds. To recon- cile this issue, and to understand the implications of constraints on probabilistic models, we take the approach of formalizing the mechanisms by which cognitive and neural processes could approximate Bayesian inference. Specifically, we show that an associative memory system using sparse, distributed represen- tations can be reinterpreted as an importance sampler, a Monte Carlo method of approximating Bayesian inference. This ca- pacity is illustrated through two case studies: a simple letter reconstruction task, and the classic problem of property induc- tion. Broadly, our work demonstrates that probabilistic mod- els can be implemented in a practical, distributed manner, and helps bridge the gap between algorithmic- and computational- level models of cognition. Keywords: Bayesian inference, importance sampling, rational process models, associative memory models, sparse distributed memory Introduction Probabilistic models of cognition can be used to explain the complex inductive inferences people make every day, such as identifying the content of images or learning new con- cepts from limited evidence (Griffiths, Chater, Kemp, Per- fors, & Tenenbaum, 2010; Tenenbaum, Kemp, Griffiths, & Goodman, 2011). However, these models are typically for- mulated at what Marr (1982) called the computational level, focusing on the abstract problems people have to solve and their ideal solutions. As a result, they explain why people behave the way they do, rather than how cognitive and neu- ral processes support these behaviors. This approach is thus quite different from previous work on modeling human cog- nition, which focused on Marr’s algorithmic and implemen- tation levels, and has been criticized because it seems to im- ply that human minds and brains need to solve intractable computational problems and use structured representations (Gigerenzer & Todd, 1999; McClelland et al., 2010). Understanding the actual commitments that computational-level accounts of human cognition based on probabilistic models make at the algorithmic and im- plementation level requires considering how these levels of analysis could be bridged (Griffiths, Vul, & Sanborn, 2012). Identifying specific cognitive algorithms and neural architectures that can approximate Bayesian inference is a key step towards knowing whether it really poses an intractable problem for human minds, or whether structured representations need to be used to implement models that involve structured probability distributions. In this paper, The associative memory that we use to approximate Bayesian inference implements a Monte Carlo algorithm known as importance sampling. Previous work has shown that this algorithm can be implemented in a common psycho- logical process model – an exemplar model (Shi, Griffiths, Feldman, & Sanborn, 2010). Shi and Griffiths (2009) further demonstrated that importance sampling can be implemented with a radial basis function neural network. However, this neural network used a localist representation, in which each hypothesis considered by the model had to be represented with a single neuron – a “grandmother cell. ” While this might be plausible for modeling aspects of perception in which a wide range of neurons prefer specific stimuli, it becomes less plausible for modeling complex cognitive tasks in which hy- potheses correspond to structured representations. For exam- ple, having separate neurons for each concept or causal struc- ture we consider seems implausible. We demonstrate that an associative memory that uses distributed representations – specifically, sparse distributed memory (SDM) (Kanerva, 1988, 1993) – can be used to ap- proximate Bayesian inference through importance sampling. The underlying idea is simple: we use the associative mem- ory to store and retrieve exemplars, allowing us to build on the equivalence between exemplar models and importance sam- pling. The critical advance is that this is done using dis- tributed representations, meaning that arbitrary hypotheses can be represented, and arbitrary distributions of exemplars encoded by a single architecture. We show that the SDM nat- urally implements one class of Bayesian models, and explain how to generalize it to implement a broader range of models. The plan of the paper is as follows. First, we give a brief overview of performing Bayesian inference with importance sampling and summarize how the sparse distributed memory system is implemented. Next, we formalize how importance sampling can be performed using a SDM. We provide two case studies drawn from existing literature in which we use the SDM to approximate existing Bayesian models. The first case study is a simple example involving reconstructing En- glish letters from noisy inputs, and the second is a more so- phisticated model of property induction. We conclude the pa- per with a discussion of implications and future directions.