Analogical Reinforcement Learning

Analogical Reinforcement Learning James M. Foster & Matt Jones james.m.foster@colorado.edu, mcj@colorado.edu University of Colorado, Department of Psychology & Neuroscience Boulder, CO 80309 USA Abstract learning. Conversely, the prediction-error signals from RL can be used to guide induction of new higher-order relational concepts. Thus we propose there exists a computationally powerful synergy between analogy and RL. The simulation experiment reported here supports this claim. Because of the strong empirical evidence for each of these mechanisms taken separately, we conjecture that the brain exploits this synergy as well. Research in analogical reasoning suggests that higher-order cognitive functions such as abstract reasoning, far transfer, and creativity are founded on recognizing structural similari- ties among relational systems. Here we integrate theories of analogy with the computational framework of reinforcement learning (RL). We propose a computational synergy between analogy and RL, in which analogical comparison provides the RL learning algorithm with a measure of relational similar- ity, and RL provides feedback signals that can drive analogical learning. Initial simulation results support the power of this approach. Keywords: Analogy; Reinforcement Learning; Schema In- duction; Similarity; Generalization Analogy Research in human conceptual knowledge representation has shown that concepts are represented not just as distributions of features (cf. Nosofsky, 1986; Rosch & Mervis, 1975) but as relational structures. This relational knowledge includes both internal structure, such as the fact that a robin’s wings allow it to fly (Sloman et al., 1998), as well as external structure, such as the fact that a dog likes to chase cats (Jones & Love, 2007). Theories of analogical reasoning represent relational knowl- edge of this type in a predicate calculus that binds objects to the roles of relations, for example CHASE ( DOG , CAT ). Ac- cording to these theories, an analogy between two complex episodes (each a network of relations and objects) amounts to recognition that they share a common relational structure (Gentner, 1983; Hummel & Holyoak, 2003). At a more mechanistic level, the dominant theory of anal- ogy is structural alignment (Gentner, 1983). This process involves building a mapping between two episodes, mapping objects to objects and relations to relations. The best map- ping is one that maps objects to similar objects, maps rela- tions to similar relations, and most importantly, satisfies par- allel connectivity. Parallel connectivity means that, when- ever two relations are mapped to each other, the objects fill- ing their respective role-fillers are also mapped together. An example is shown in Figure 1. Parallel connectivity is sat- isfied here because, for each mapped pair of ATTACK rela- tions (red arrows), the objects filling the ATTACKER role are mapped together (knight is mapped to queen), and the objects filling the ATTACKED role are also mapped together (rook to rook and king to king). Thus structural alignment con- stitutes a (potentially partial or imperfect) isomorphism be- tween two episodes, which respects the relational structure that they have in common. Importantly, if the search for a mapping gives little emphasis to object-level similarity (as opposed to relation-level similarity and parallel connectivity), then structural alignment can find abstract commonalities be- tween episodes having little or no surface similarity (i.e., in terms of perceptual features). We propose structural alignment is critical to learning of Introduction The goal of the present work is to develop a computational understanding of how people learn abstract concepts. Pre- vious research in analogical reasoning suggests that higher- order cognitive functions such as abstract reasoning, far trans- fer, and creativity are founded on recognizing structural sim- ilarities among relational systems (Doumas et al., 2008; Gen- tner, 1983; Hummel & Holyoak, 2003). However, we argue a critical element is missing from these theories, in that their operation is essentially unsupervised, merely seeking patterns that recur in the environment, rather than focusing on the ones that are predictive of reward or other important outcomes. Here we integrate theories of analogy with the computa- tional framework of reinforcement learning (RL). RL offers a family of learning algorithms that have been highly success- ful in machine learning applications (e.g., Bagnell & Schnei- der, 2001; Tesauro, 1995) and that have neurophysiological support in the brain (e.g., Schultz et al., 1997). A shortcom- ing of RL is that it only learns efficiently in complex tasks if it starts with a representation (i.e., a means for encoding stimuli or states of the environment) that somehow captures the crit- ical structure inherent in the task. We formalize this notion below in terms of similarity-based generalization (Shepard, 1987) and kernel methods from statistical machine learning (Shawe-Taylor & Cristianini, 2004). In other words, RL re- quires a sophisticated sense of similarity to succeed in real- istically complex tasks. Psychologically, the question of how such a similarity function is learned can be cast as a question of learning sophisticated, abstract representations. This paper proposes a computational model of analogical RL, in which analogical comparison provides the RL learning algorithm with a measure of relational similarity, and RL pro- vides feedback signals that can drive analogical learning. Re- lational similarity enables RL to generalize knowledge from past to current situations more efficiently, leading to faster