Duality Between Feature and Similarity Models , Based on the Reproducing-Kernel Hilbert Space

There are two longstanding theoretical approaches to learning and concept representation, one based on features and one based on similarity. The feature approach has its roots in associative learning (e.g., Pavlov, 1927) and the idea that learning involves acquiring associations from individual cues to consequential outcomes or responses. Formal models of this learning process assume that any stimulus is decomposable into a set of features, and that learning involves adjusting the associative strength of each feature (Estes, 1950; Rosenblatt, 1958). More modern versions of these models posit learning rules whereby features compete to make predictions (Rescorla & Wagner, 1972), a mechanism that famously explains blocking (Kamin, 1968) and other forms of cue competition in category learning (Gluck & Bower, 1988). The similarity approach makes a radically different assumption, that learning is centered on whole stimuli rather than on substituent features. This view has its roots in experimental work on stimulus generalization (Guttman & Kalish, 1956) and the idea that learning can be characterized by the degree to which knowledge acquired about one stimulus is transferred or generalized to another. Formal models of similarity-based learning assume that responses to new stimuli are determined by direct generalization from previously encountered stimuli (Nosofsky, 1986; Shepard, 1987). Learning per se is based on acquisition of new exemplars (Nosofsky, 1986) or on learning configural associations from whole stimuli to outcomes (Kruschke, 1992; Pearce, 1987, 1994). Similarity-based models have shown great success in predicting transfer performance on novel stimuli (Nosofsky, 1992), particularly in cases where behavior is not well captured by an additive function of stimulus features (Medin & Schaffer, 1978; Pearce & Bouton, 2001). The debate between featureand similarity-based learning has a long history in both animal conditioning, as a debate between elemental and configural models, and in human category learning, as a debate between examplar models and prototype or connectionist models. The debate seems to get at fundamental questions regarding the nature of knowledge and psychological representation. Are conceptual and causual knowledge encoded in associations among individual features (cues), or in memory for specific instances (exemplars)? Is a stimulus fundamentally represented in an analytic fashion, in terms of its values on various features, or holistically, in terms of its similarity to other stimuli? It seems natural to believe that one of these views must be more correct than the other, and that clever experimental and computation research should eventually be able to determine which set of principles governs the brain’s operation. However, recent theoretical developments in machine learning suggest that the difference between feature and similarity models is not as fundamental as psychologists have believed. Instead, the principles of these two approaches turn out to be compatible, via a deep mathematical connection. This connection is embodied in the kernel framework, a mathematical system that has been used to formulate a variety of powerful machine-learning algorithms for classification and prediction (Schölkopf & Smola, 2002). Under the kernel framework, simple linear algorithms, such as linear regression or the support vector machine, are made much more flexible by first projecting the stimulus set into a high (usually infinite) dimensional vector space, and then applying the algorithm in that new space (Shawe-Taylor & Christianini, 2004). The vector space is called a reproducing-kernel Hilbert space (RKHS), and it is generated by a kernel function that assigns a real number to every pair of stimuli in the original space. In principle, learning and optimization are done in the RKHS, which is what gives kernel methods their great flexibility. However, in practice all necessary