Superposition catastrophe 1 Running head : SUPERPOSITION CATASTROPHE PDP Networks Learn Local ( Grandmother Cell ) Representations in Order to Code for Multiple Things at the Same Time

One potential limitation with distributed representation is that it is difficult to co-activate multiple things at the same time over the same set of processing units. That is, a superposition of co-active distributed representations results in a blend pattern that is ambiguous; the socalled superposition catastrophe. Thus it is striking that M. M. Botvinick and D. C. Plaut (2006) developed a PDP model of short-term memory that recalls lists of letters based on the superposition of distributed letter codes. Their finding suggests that the distributed representations can solve the superposition catastrophe. However we show that the model’s success does not mitigate against this constraint. Under the appropriate training conditions a version of their model can indeed solve (avoid) the superposition catastrophe, but only by learning localist as opposed to distributed representations. Given that many cognitive systems need to code multiple things at the same time, the pressure to learn localist (“grandmother cell”) representations is widespread. Superposition catastrophe 3 PDP Networks Learn Local (Grandmother Cell) Representations in Order to Code for Multiple Things at the Same Time Neural networks can code for information in two qualitatively different ways. Some models rely on localist representations, such that words, objects, faces, and lexical concepts are coded distinctly, with their own dedicated representations. For example, in localist models of written word identification, the words dog and log are coded with distinct orthographic units, and a word is identified when a single unit (e.g., dog) is activated beyond some threshold (e.g., McClelland & Rumelhart, 1981). Other models rely on distributed representations, such that the words dog and hog are coded as patterns of activation over collections of units, with each unit involved in coding multiple words (e.g., Seidenberg & McClelland, 1989). In fact, various different types of distributed representations can be distinguished, ranging from dense distributed representations in which many hidden units are co-activated in response to a single input and each hidden unit is involved in coding many different things, to sparse distributed coding in which each input activates relatively few units, and each unit is involved in coding relatively few things (but more than one). Nevertheless all versions of distributed coding share one defining feature; that is, it is not possible to interpret the activation of a single unit unambiguously. The debate regarding the relative merits of localist and distributed coding has taken a variety of forms, including their relative biological plausibility (e.g., Bowers, 2009, 2010ab; Plaut & McClelland, 2010ab; Quian Quiroga & Kreiman, 2010ab), their empirical successes (e.g., Coltheart, Rastle, Perry, Langdon, & Ziegler, 2001; Seidenberg & Plaut, 2006), and their computational capacities (e.g., Botvinick & Plaut, 2009ab; Bowers, Damian, & Davis, 2009ab, Page, 2000; Plaut & McClelland, 2000). In the present paper we focus on the computational capacity of the distributed representations typically learned in PDP models, and argue that they are ill suited to support the co-activation of multiple items at the same time over the same set of units. That is, we claim that these distributed representations cannot overcome the superposition catastrophe, contrary to some recent findings that suggest otherwise (Botvinick & Plaut, 2006). We argue that the functional requirement to co-activate multiple things at the same time provides a computational pressure to develop representations that respond highly selectively to inputs. Indeed, we show that PDP models sometimes learn local (“grandmother cell”) representations in response to this constraint. Catastrophic interference There is already one well known computational limitation of the dense distributed representations often learned in PDP models; namely these representations are poor at supporting rapid one-trial learning, as required for episodic memory. The problem is that each unit in a dense distributed representation is involved in coding many things, and as a consequence, learning something new impacts on pre-existing knowledge (specifically, knowledge coded with overlapping units and connection weights). The greater the learning the greater the impact on pre-existing knowledge, and in the case of rapid one-trial learning, new knowledge can erase prior knowledge, a phenomenon known as catastrophic interference (McClosky & Cohen, 1989; Ratcliff, 1990), or the stability-plasticity dilemma (Grossberg, 1976). According to McClelland, McNaughton, and O’Reilly (1995), the brain’s solution to this problem was to develop sparse coding in the hippocampus and related structures, with relatively few units active at any one point, and each unit involved in coding relatively few things. These sparse codes are not considered local (each unit would be involved in coding for more than one thing), but nevertheless, the solution to catastrophic interference is to reduce the level of distribution. In this way, large changes in connections weights (in response to new learning) have minimal impact on prior knowledge. Superposition catastrophe 4 However, this solution is itself limited. PDP models generalize on the basis of new inputs activating hidden units that are involved in coding old knowledge, and this overlap is often reduced when knowledge is coded sparsely. This makes the sparse codes in the hippocampus a poor medium for generalizing. For this reason, McClelland et al. (1995) argue that the brain relies on both dense and sparse distributed coding schemes: dense distributed representations in the cortex that support various forms of generalization required for perception, language, and semantic memory, and sparse distributed codes in the hippocampus (and related structures) for episodic memory. This dual solution to fast learning and generalization is the so-called complementary learning systems hypothesis. The key claim we make here is that there is yet another computational constraint that undermines the use of dense distributed representations in the cortex as well. That is, the dense distributed representations learned in PDP networks are not only poor at fast learning, but also poorly suited to the task of co-activating multiple things at the same time. Accordingly, the challenge is to develop neural networks that can both generalize and co-activate multiple things. We show that PDP networks that learn localist representations of letters in response to the superposition constraint can indeed generalize to novel words. The Superposition Catastrophe PDP models support generalization in a variety of domains (e.g., reading and semantics), but until recently, there have been no demonstrations that these models can co-activate multiple things at the same time over the same set of units. Given that various cognitive systems support the co-activation of multiple items at the same time (perhaps 4 +/-1; Cowan, 2001), this is a striking omission. Particularly so given that it has been claimed that distributed representations are ill suited for supporting this function, due to the superposition catastrophe (von der Malsburg, 1986). The superposition catastrophe refers to the observation that co-activating items in a distributed system leads to blend patterns that are ambiguous with regards to what produced the blend in the first place. For example, consider Figure 1, adapted from Page (2000), that depicts distributed patterns for the names Paul, John, George, Ringo, Mick, and Keith. The patterns are distributed given that each name is coded as a pattern of activation of two of four units, and the identity of a person cannot be determined by the activation of a single unit. Although the identity of a name can be determined based on the pattern of activation over the four units, the coding scheme is ambiguous when more than one person is coded at the same time over the same units. For example, the combination of MICK and KEITH co-activates all four units, but so do JOHN and GEORGE. The tempting conclusion is that it is difficult to code for more than one thing at a time over a given set of processing units when relying on distributed representations. By contrast, ambiguity is eliminated with localist coding schemes. For example, as can be seen in Figure 2, if localist Mick and Keith units are co-active, the constituent patterns that produced the blend could only be Mick and Keith. Bowers (2002) used the superposition catastrophe as an argument in favor of localist coding models. The ambiguity associated with a superposition of dense distributed representations is in fact worse than the example above. In Figure 1, the ambiguity is the product of blending together two familiar names (e.g., Mick and Keith), but blends might just as well be produced by combinations of novel inputs. If novel inputs are free to contribute to blends, then there is an infinity of different combinations of novel constituents that will produce the same blend. In this sense the superposition catastrophe is an ill-posed problem; it is logically impossible to determine the constituent patterns that produced a given blend. This would seem to rule out dense distributed representations as a medium for supporting cognitive processes that deal with more than one thing at a time (that is, most of cognition). Nevertheless, it is important to note that the brain faces and often solves other ill posed problems. The classic example is the so-called inverse problem in vision, in which an infinite Superposition catastrophe 5 number of possible shapes in the 3D world are consistent with a given 2D image projected on the retina. The conclusion is not that we cannot perceive in 3D, but rather, that the visual system needs to adopt biases about which 3D shapes are most likely to have produced a given (ambiguous) 2D image on the retina. The biases c