Trade-off Between Capacity and Generalization in a Model of Memory

Trade-off Between Capacity and Generalization in a Model of Memory Guy Tannenbaum (guy tannenbaum@yahoo.com) and Yehezkel Yeshurun (hezy@post.tau.ac.il) School of Computer Science, Tel Aviv University Ramat Aviv, Tel Aviv 69978, Israel Shimon Edelman (se37@cornell.edu) Department of Psychology, Cornell University Ithaca, NY 14853, USA Abstract understanding of memory, such as that of Minsky (1985), have typically been only partially implemented (Hearn, 2001). An explicit computational model designed to pro- vide both storage and generalization has been recently developed by Mueller and Shiffrin, but its reported eval- uation (Mueller, 2006; Mueller & Shiffrin, 2006) seems to be qualitative rather than quantitative. Our goal in the present study has been to investigate the emergence of a trade-off between capacity and gen- eralization under conditions that are as general as pos- sible. To that end, we chose to focus on implementing a functional (rather than neuromorphic) computational model of pattern storage and generalization, which ex- tends that of Moll and Miikkulainen (1997). This allows us to relate our results to existing methods and findings regarding memory capacity. Although computational considerations suggest that a resource-limited memory system may have to trade off capacity for generalization ability, such a trade-off has not been demonstrated in the past. We describe a sim- ple model of memory that exhibits this trade-off and describe its performance in a variety of tasks. Keywords: memory; computational model; capacity; generalization; trade-off. Introduction Because the probability of a cognitive agent encounter- ing precisely the same stimulus twice is infinitesimally small, memories of past experiences are only useful for guiding future behavior insofar as they can be general- ized so as to apply to new variations on familiar themes. Intuitively, it would seem that in a memory system bet- ter generalization would have to come at the expense of reduced capacity. Indeed, the famous mnemonist patient studied by Luria (1968), whose memory capacity seemed practically unlimited, was oblivious even to simple pat- terns in the memorized items. From the functional standpoint, this intuition can be related to the distinction commonly made between “sim- ple” or episodic memory (pertaining to statistically rare but important events, such as who did what to whom), where capacity is the key goal, and conceptual memory (pertaining to statistically redundant patterns of events in the environment), where generalization between “sim- ilar” items is crucial (Merker, 2004). Because these memory functions, along with the many others in the human brain (Rolls, 2000), are subject to certain con- straints on the available resources, one expects memory systems to exhibit a trade-off between capacity and gen- eralization. Surprisingly little computational work has been de- voted to testing this prediction. Does the expected trade-off arise generically, in any resource-limited mem- ory system? Despite important early insights into the computational underpinnings both of simple and of con- ceptual memory (Brindley, 1969; Marr, 1969, 1970), sub- sequent “connectionist” memory models, such as those of Hopfield (1982) or Kanerva (1988), did not consider this question, in part because they had not been intended to provide generalization capabilities. At the same time, the more comprehensive theoretical frameworks for the The computational framework Our model operates on vectors of positive integers. Its building blocks are “neurons” with real-valued activa- tion, connected via real-valued synaptic weights. Perfor- mance is measured as a function of the resources avail- able to the system, and of how they are allocated be- tween the different aspects of each task. Capacity We measure memory capacity by assessing the model’s ability to recognize previously encountered patterns. Given a query pattern, the model returns the probability of having encountered it before. The results are plotted in the form of a receiver operating characteristic (ROC). Capacity is then defined as the number of patterns that the model can memorize while maintaining a given area under the ROC curve. This metric has a natural psy- chological interpretation: human memory is often faced with the task of deciding whether or not a pattern has been seen before (Koriat, Goldsmith, & Pansky, 2000). Alternatively, capacity can be defined in terms of com- pletion of partial patterns (Moll & Miikkulainen, 1997). Generalization Capturing an artificial hierarchical taxonomy. A basic task that involves generalization is similarity esti- mation: the memory representation of some structured collection of objects, such as items drawn from a taxo-