Toward a New Grand Theory of Development? Connectionism and Dynamic Systems Theory Reconsidered

I discuss a connectionist model, based on Elman’s (1990, 1991) Simple Recurrent Network, of the acquisition of complex syntactic structure. While not intended as a detailed model of the process children go through in acquiring natural languages, the model helps clarify concepts that may be useful for understanding the development of complex abilities. It provides evidence that connectionist learning can produce stage-wise development. It is consistent with prior work on connectionist models emphasizing their capability of computing in ways that are not possible within the symbolic paradigm (Siegelmann, 1999). On the other hand, it suggests that one mechanism of the symbolic paradigm (a pushdown automaton) may be identified with an attractor of the learning process, thus providing a useful bearing point in the complex landscape of dynamical computation. Thus, there is a menagea-trois: a correspondence between dynamical systems theory and symbolic computation theory is embodied in a connectionist network. Introduction This paper adds its voice to those that have pointed out that connectionist models are a subclass of dynamical systems. They are a particularly interesting subclass because of their psychological plausibility. Moreover, they have a valuable trait: one can make connectionist models that behave in psychologically plausible ways but which are, at least In addition to the very helpful interactions at the conference that prompted this book, I had enlightening discussions with James Dixon, James Magnuson, Jay Rueckl, and Dalia Terhesiu. DYNAMICAL INSIGHT INTO STRUCTURE IN CONNECTIONIST MODELS 2 initially, inscrutable. Inscrutability might seem like a negative point. But one can sometimes learn things by striving to see into it. Dynamical systems theory is helpful in this regard: it provides conceptual constructs (e.g., trajectories, basins, attractors, bifurcations) and analytical methods (e.g., phase-space portraits, linearization) that can help one understand relationships and causes in these psychologically relevant models. At this point, it is helpful to mention a third element: symbolic computation theory. By symbolic computation theory, I mean theories which describe behavior using discrete symbols and well-formedness rules. Observed behaviors are modeled as allowable symbol patterns under the well-formedness rules. The grammatical theories of natural language proposed by Generative Grammar are examples of this kind of theory (e.g., Chomsky, 1957). The symbolic approach has generally been treated as an enemy by both connectionists and ecological psychologists, including many dynamical-systems oriented ecological psychologists. While I agree with the rejection of symbolic models as a sufficient foundation for the science of mind-world systems, I argue here that symbolic models are nevertheless particularly relevant to cognition, and in fact, to the relationship between the connectionist and dynamical approaches. In a nutshell, symbolic models correspond to certain idealized behaviors which connectionist models can exhibit and which influence learning the way an attractor can be said to influence the trajectory of a dynamical system, even though realistic connectionist systems rarely, if ever, spend time at these attractors. I’ll make this more concrete by talking about one major domain: development. Several people have suggested that the dynamical notion of a phase transition is particularly relevant to development (Thelen & Smith, 1994; van der Maas & Molenaar, 1992; van Geert, 1998). In the sense intended here, a phase transition is a qualitative change in the behavior of a system that is associated with a monotonic and continuous change of a parameter, called a control parameter. Phase transitions seem potentially relevant to modeling developmental stages because they correspond to situations in which behavior changes subtly and quantitatively for a long period of time and then suddenly changes dramatically and qualitatively. Indeed, the researchers mentioned above have pointed out plausible resemblances between developmental data and various mathematical models of phase transitions. Here, I go along this path and then make an additional suggestion: developmental phase transitions may involve a shift from a phase in which well-formedness rules fail to characterize the structure of behavior to a phase in which these rules constitute a good approximation of the behavior. Thus, the account treats the symbolic models as relevant to characterizing stages, but not sufficient for characterizing all the states people go through in the process of growing up. What is the evidence for these claims? It takes the form of a connectionist model called a Fractal Learning Neural Network (FLNN). I want to make clear at the outset that I offer no measured developmental data to motivate this model at all. Instead, one might think of this work as an exploration of abstract information-structure. For two reasons, I believe that the results may be of interest to developmentalists: 1. The model is a connectionist learning model and thus has relatives which have made plausible empirical predictions about many domains, including development; 2. The model helps a person think clearly about an appealing alternative to some current theories of development (e.g., maturational explanations for stages, superficial nativist accounts). The model’s account is appealing because (a) it uses a simple learning mechanism and thus corresponds to a theory that links development and learning, (b) it predicts stage-wise behavior with progressive quantitative DYNAMICAL INSIGHT INTO STRUCTURE IN CONNECTIONIST MODELS 3 changes occurring within the stages, a phenomenon also observed in development, (c) it derives the phase transitions from the learning, thus predicting, rather than stipulating, the stages, (d) it shows highly structured, domain-specific behavior arising from a mechanism which is capable of a wide variety of behaviors, not all of which are highly structured, thus, in some sense, deriving the structure, (e) it generalizes beyond its input in a way that is at least partially consistent with human generalization. I will describe the model and then return to these points. Description of the Model Background Elman (1990, 1991) trained a network, which he dubbed the “Simple Recurrent Network”, to process sequences of symbols, including sequences of words. The network (Figure 1) has three layers of units (input, hidden, and output) as well as a context layer, which stores a copy of the hidden unit activations from the previous time step. The connections feed forward from input to hidden to output and from context to hidden. The copying operation associated with the context layer is a kind of recurrence, and it permits the network to encode information about past inputs, allowing it to keep track of the temporal dependencies which are crucial to syntactic processing. In one experiment, Elman trained his network on a corpus of words approximating the syntactic properties of English. The corpus included singular and plural nouns, as well as center-embedded relative clause structures (1) (articles like “the” and a were left out for simplicity). (1) dogs/dog who girls chase run/runs. In an example like (1), the network must store the number of the head noun of the subject of the sentence (“dog” or “dogs”) during the time that it processes the relative clause (“who girls chase”) in order to properly predict the number of the main verb (“run” vs. “runs”). Elman used a single-on-bit (“one-hot”) encoding for each word on the input layer and on the output layer. The codes of the words of the corpus were presented in order, one at a time, on the input layer of the network. The network was trained (using the sum-squared error cost function and backpropagation of the error signal through time Rumelhart, Hinton, & Williams, 1986; Pearlmutter & MacDonald, 1995) on the task of predicting the next word at each point in time. Because the sequence of sentences in the corpus was approximately random, the network was not able to guess specifically what word was going to appear at any point in time. However, it could pick up on stable syntactic properties of the language. In particular, once the network was fully trained, the activation of each output unit approximated the probability that the word corresponding to that output unit would occur next in the corpus. For example, after having encountered the word “dog” at the beginning of a sentence, output activation was concentrated mostly on the singular verbs and the word “who”. While such local co-occurrence tendencies are fairly easy to model (e.g., with Markov processes), some kinds of longer distance dependencies, including those arising in center-embedded structures like (1), are more difficult. Thus, it is significant that the network was also successful at predicting the number on the verbs in relative clause structures like (1). DYNAMICAL INSIGHT INTO STRUCTURE IN CONNECTIONIST MODELS 4 Input cat cats leaps . leap Context . Hidden . Output