One-Shot Learning with Bayesian Networks

One-Shot Learning with Bayesian Networks Andrew L. Maas (amaas@andrew.cmu.edu) Computer Science Department & Center for the Neural Basis of Cognition Carnegie Mellon University Charles Kemp (ckemp@cmu.edu) Department of Psychology & Center for the Neural Basis of Cognition Carnegie Mellon University (b) (a) Abstract Nationality British British Brazilian Brazilian Spanish Spanish Randerian Randerian Humans often make accurate inferences given a single exposure to a novel situation. Some of these inferences can be achieved by discovering and using near-deterministic relationships between attributes. Approaches based on Bayesian networks are good at discovering and using soft probabilistic relationships between attributes, but typically fail to identify and exploit near-deterministic relationships. Here we develop a Bayesian network approach that overcomes this limitation by learning a hyperparameter for each distribution in the network that specifies whether it is non-deterministic or near-deterministic. We apply our approach to one-shot learning problems based on a real-world database of immigration records, and show that it outperforms a more standard Bayesian network approach. Language English English Portuguese Portuguese Spanish Basque Randerian Hair Red Brown Blonde Brown Brown Black Blonde Language Randerian English Hair Color Blonde Brown Figure 1: Randeria one-shot learning problem. (a) Af- ter meeting people from several different countries, you might discover that people from the same country tend to speak the same language. (b) Discovering the pattern in (a) supports one-shot learning about people from a new country. After observing a single Randerian, you might have strong expectations about the language spo- ken by a subsequent Randerian, but weak expectations about her hair color. Keywords: Machine Learning; One-Shot Learning; Concepts and Categories; Bayesian Modeling Introduction Humans are able to discover and exploit relationships between attributes (e.g. nationality and language) and between attribute values (e.g. Brazilian and Portuguese) (Davies & Russell, 1987). Some relationships are near- deterministic, including the relationship between birth country and native language. We know, for example, that two individuals born in the same country are very likely to have the same mother tongue, and we know in particular that individuals born in Brazil are very likely to speak Portuguese. Other relationships are probabilis- tic, including the relationship between hair color and eye color. We know that these attributes tend to be related, and we know about specific relationships between values of these attributes (blondes often have blue eyes). Suppose, for example, that after meeting several peo- ple from various countries, you meet a single person from Randeria, a country that is completely new to you. You observe that the person has blonde hair and speaks Ran- derian. Based on this single example, you may be very confident that the next Randerian you meet will speak the same language, but less confident that this second Randerian will also have blonde hair. Figure 1(a) shows a schematic representation of the observed data, and Figure 1(b) shows conditional distributions that capture our expectations about the language and hair color of the second Randerian. The Randeria problem just in- troduced is a special case of the more general problem of one-shot learning (Fei-Fei, Fergus, & Perona, 2003). Here we describe and evaluate a probabilistic model that can handle one-shot learning problems similar to the Randeria problem. One-shot learning has been previously considered in the psychological literature. One prominent line of work has focused on “fast mapping” in word learning (Carey & Bartlett, 1978; Smith, Jones, Landau, Gershkoff-Stowe, & Samuelson, 2002). Empirical studies of word learn- ing have documented that children are able to learn the meaning of some new words given a single training exam- ple and researchers have developed formal models (Col- unga & Smith, 2005; Kemp, Perfors, & Tenenbaum, 2007) that help to explain this ability. Our approach grows out of this literature, and the work we describe builds on the hierarchical Bayesian model presented by Kemp et al. (2007). Hierarchical Bayesian models (Gel- man, Carlin, Stern, & Rubin, 2003) can include rep- resentations at multiple levels of abstraction, and help to explain how humans acquire abstract knowledge that supports rapid or one-shot learning given exposure to a novel situation. Our hierarchical Bayesian approach is built on top of a standard method for learning Bayesian networks, also