Learning nonadjacent dependencies in thought, language, and action: Not so hard after all.

Learning nonadjacent dependencies in thought, language, and action: Not so hard after all… Jon A. Willits (jwillits@indiana.edu) Indiana University, Department of Psychological and Brain Sciences, 1101 E. 10 th St. Bloomington, IN 47405 USA Abstract Learning to represent hierarchical structure and its nonadjacent dependencies (NDs) is thought to be difficult. I present three simulations of ND learning using a simple recurrent network (SRN). In Simulation 1, I show that the model can learn distance-invariant representations of nonadjacent dependencies. In Simulation 2, I show that purely localist SRNs can learn abstract rule-like relationships. In Simulation 3, I show that SRNs exhibit facilitated learning when there are correlated perceptual and semantic cues to the structure (just as people do). Together, these simulations show that (contrary to previous claims) SRNs are capable of learning abstract and rule-like nonadjacent dependencies, and show critical perceptual- and semantics-syntax interactions during learning. The studies refute the claim that neural networks and other associative models are fundamentally incapable of representing hierarchical structure, and show how recurrent networks can provide insight about principles underlying human learning and the representation of hierarchical structure. Keywords: hierarchical structure; recurrent connectionist networks; nonadjacent dependencies Background Human concepts, languages, goals, and patterns of action are all describable in terms of complex hierarchical structures, but our experience of them as inputs, and our production of them as outputs, is often arranged in linear strings that unfold over time. A necessary consequence of this transformation of complex structure into linear strings is that most human knowledge involves many nonadjacent dependencies, where one element predicts another element, but at a distance. These nonadjacent dependencies, whether in thought, language, or action, enormously expand the computational complexity of representing the structure of the world. In several subfields of cognitive science, difficulty learning and representing nonadjacent dependencies has generated considerable theoretical controversy. In linguistics, the limitation of simple associative structures has been a cornerstone of arguments for abstract syntactic structures (Chomsky, 1957). In cognitive psychology, researchers argued that associative mechanisms cannot learn the vast range of nonadjacent dependencies in the world, and thus rule- based representations are necessary for human cognition (Bever et al., 1968). In early artificial intelligence, arguments about the limitations of associative systems led to a focus on symbolic, rule- based systems (Newell & Simon, 1961). However, recent research has questioned the need for rule-based representations of nonadjacent structure. A number of studies have demonstrated or modeled simple learning of nonadjacent structure in memory (Cleeremans & McClelland, 1991), goals and event structure (Botvinick & Plaut, 2004; visual sequences (Fiser & Aslin, 2002), and artificial grammars using linguistic stimuli (Gomez, 2002; Newport & Aslin, 2004). These results have changed the nature of the debate concerning the extent to which knowledge of nonadjacent dependencies requires a rule-based or an association-based explanation. Although there are many specific examples of learning or failing to learn in particular situations, what is lacking is a general account of nonadjacent dependency learning. As a result, the many subfields of cognitive science (such as linguistics, cognitive psychology, and artificial intelligence) continue working on the problem separately, without a clear theory or explanation for some of the most foundational human behaviors. The current work aims to make progress toward a general account by examining whether a fairly simple neural network model, the simple recurrent network (SRN; Elman, 1990) can provide a general model of nonadjacent dependency learning. An SRN was used because previous research (Botvinick & Plaut, 2006; Cleeremans & McClelland, 1991; Elman, 1991) suggests that SRNs and other recurrent networks are capable of learning nonadjacent structure. However, there is controversy about whether they can serve as general solution for all cases, especially those involving abstract, rule-like relationships (Marcus, 2000) or complex interactions between structure and meaning (Fodor & Pylyshyn, 1988). In the service of testing the viability of SRNs, the current work had two distinct sub-goals. First, to be a general model of nonadjacent dependency learning, SRNs ought to be able to learn nonadjacent dependencies of the types that exist in the natural