Comparing the inductive biases of simple neural networks and Bayesian models

Comparing the inductive biases of simple neural networks and Bayesian models Thomas L. Griffiths (tom griffiths@berkeley.edu) Joseph L. Austerweil (joseph.austerweil@gmail.com) Vincent G. Berthiaume (vberthiaume@berkeley.edu) Department of Psychology, University of California, Berkeley, CA 94720 USA Abstract Understanding the relationship between connectionist and probabilistic models is important for evaluating the compati- bility of these approaches. We use mathematical analyses and computer simulations to show that a linear neural network can approximate the generalization performance of a probabilis- tic model of property induction, and that training this network by gradient descent with early stopping results in similar per- formance to Bayesian inference with a particular prior. How- ever, this prior differs from distributions defined using discrete structure, suggesting that neural networks have inductive bi- ases that can be differentiated from probabilistic models with structured representations. Keywords: Bayesian modeling, connectionism, inductive bi- ases, property induction to be found by training neural networks by gradient descent comparable to those produced by Bayesian inference (that is, are the inductive biases of these approaches related)? Finally, how compatible are the inductive biases of neural networks with those of structured probabilistic models? We provide positive answers to the first two questions, showing that a simple neural network can always approximate a probabilis- tic model of property induction, and that training this network using a gradient descent algorithm is similar to Bayesian in- ference with a particular prior distribution. However, we also show that there remains a significant difference between this prior and distributions based on discrete representations. Introduction Mathematical analysis Cognitive scientists use different mathematical formalisms to model human cognition. Understanding the relationships between these approaches is critical to resolving questions about the nature of the mind. Recently, researchers have de- bated whether probabilistic or connectionist models of cog- nition provide better prospects for making progress in cogni- tive science (Griffiths, Chater, Kemp, Perfors, & Tenenbaum, 2010; McClelland et al., 2010). One of the key issues in this debate is that many probabilistic models are defined in terms of structured, discrete representations, while connec- tionist models use continuous, graded representations that can mimic discrete structure when needed. A possible resolution would be to view probabilistic and connectionist models as lying at different levels of analysis (Marr, 1982), with neural networks a continuous approximation to Bayesian inference over discrete representations. However, this requires estab- lishing whether such an approximation is possible. To explore this issue, we use the problem of property in- duction as a case study for investigating the relationship be- tween probabilistic models of cognition and neural networks. Property induction – inferring the properties of a novel ob- ject based on the properties of other objects – has played a key role in the debate between probabilistic and connection- ist models. An influential probabilistic model explains hu- man property induction in terms of Bayesian inference over discrete representations such as graphs and trees (Kemp & Tenenbaum, 2009), whereas a successful connectionist model explains people’s inferences via continuous representations learned by gradient descent (Rogers & McClelland, 2004). We use a combination of mathematical analysis and com- puter simulations to address three questions. First, can a prob- abilistic model with a discrete representation for a set of ob- jects be approximated by a neural network model with con- tinuous representations? Second, are the solutions that tend Our mathematical analysis focuses on comparing the model of property induction used by Kemp and Tenenbaum (2009; henceforth KT09) with a linear neural network. Setting up the problem The KT09 model assumes that we want to capture the joint distribution of the elements of continuous N-dimensional vectors x indicating the value of a single property for N ob- jects. 1 This distribution, p(x), results from a diffusion pro- cess on a graph. The diffusion process induces a multivariate Gaussian distribution on x with mean zero and covariance Σ discrete = ∆ + 2 I where ∆ is the Laplacian of the graph, being G − I for a graph with adjacency matrix G, and I is the identity matrix. Now consider a linear neural network model. 2 This model represents an observed N × M matrix (the values of M prop- erties for N objects) as the product X = YZ where X is the N × M matrix of observed objects, Y is an N × K matrix, and Z is a K × M matrix. In this model, Z is the representation of the set of properties on a hidden layer with K units (as might be encoded in the weights from an 1 This formulation is a little counter-intuitive, as the set of objects is fixed but the set of properties is left open (ie. new properties tend to be observed, rather than new objects). This differs from the most intuitive way of thinking about the problem for a neural network, in which the network is trained to predict the properties that objects have, with the set of properties fixed and the set of objects left open. 2 Neural network models typically use non-linear activation func- tions at the hidden layer. This complicates the analysis, but we hope to explore the consequences of such non-linearities in future work. We return to this point in the Discussion.