The Sensorimotor Bases of Linguistic Structure : Experiments with Grounded Adaptive Agents

This research uses grounded adaptive agents for investigating the evolutionary origins of syntactic categories, such as nouns and verbs. To analyze the sensorimotor bases of linguistic structure, the techniques of categorical perception and of synthetic brain imaging are employed. The simulation uses two different architectures for the adaptive agent’s neural controller. Analyses show that the neural processing of verbs is consistently localized in the regions of the networks that perform sensorimotor integration, while nouns are associated with sensory processing areas. The general implications of such model and of the analysis techniques for adaptive behavior and language evolution research are discussed. 1. Modeling language evolution in grounded adaptive agents Investigations on the emergence of language, both in evolutionary and developmental terms (MacWhinney 1999; Cangelosi & Parisi 2002), have greatly benefited from the use of computational models. This has resulted in an alternative approach to the nature vs. nurture (and biological vs. cultural) dilemma in language emergence. Models demonstrate that language is a complex system that emerges from intricate interactions between various biological and environmental processes. Adaptive behavior and artificial life provide useful modeling methodologies (Kirby 2002; Wagner et al. 2003) for dealing with such a complex system vision of language. In adaptive behavior models, populations of autonomous agents interact via language games to exchange information about the environment. Their coordinated communication system emerges from the direct interaction between agents. Amongst the various adaptive behavior approaches, some provide a more integrative vision of language and treat it as an integral part of the whole cognitive system. The agent’s linguistic abilities are strictly dependent on, and grounded in, other behaviors and skills. Various sensorimotor, cognitive, neural, social and evolutionary factors contribute to the emergence and establishment of communication and language. For example, in these models there exists an intrinsic link between the communication symbols (words) used by the agent and its own cognitive representations (meanings) of the perceptual and sensorimotor interaction with the external world (referents). We call this grounded adaptive agent modeling of the emergence of language. This is consistent with the psychologically-plausible theories of the grounding of language in the organism’s perceptual and action systems (Barsalou 1999; Glenberg & Kaschak in press; Joyce et al. 2003). Such an approach is in opposition to other adaptive modeling systems that view language as an independent and autonomous capability of the agent, and are subject to the symbol grounding problems (Harnad, 1990). There are various language models based on grounded adaptive agents. Some use real (or realistic) robots interacting in physical environments, while others use simulated adaptive agents. In robotic models, communication results from the dynamical interaction between the robot’s physical body, its cognitive system and the external physical and social environment. Some studies stress the grounding in sensorimotor processes, such as Marocco’s et al. (2003) model of robotic arms in a physics dynamics simulator and Vogt’s (2002) mobile robots. Other robotic models highlight the grounding through social interaction, such as Steels’ (2002) talking heads and Kaplan’s (2000) interactive toy robot. On the other hand, some studies are based on simulation adaptive agents. They model the agent and its environment with a good degree of detail upon which emergent meanings can be directly constructed. These simulation models have focused on grounding in perceptual experience (Cangelosi et al. 2000; Dyer 1994) and in cognitive representations and sensorimotor interactions (Cangelosi & Harnad 2000; Hazlehurst & Hutchins 1998). Both robotic and simulation models have been used to study the emergence of simple communication systems, and that of more complex syntactic languages. Of the models focusing on syntax, some have specifically addressed the evolutionary transition from non-compositional communication to syntactic language. This is an important issue in the field of language origins, because it can shed light on the evolutionary emergence and diversification of word categories, such as nouns, verbs and functions words (e.g. prepositions and quantifiers). This topic is also very important for studies on the ontogenetic development of language based on the emergentist/constructivist approach (Tomasello & Brooks, 1999; MacWhinney 1998). This paper will focus on the use of grounded adaptive agents for investigating the evolutionary origins of syntactic categories, such as nouns and verbs. It will use grounded adaptive agents in which neural networks control all behaviors, including sensorimotor, cognitive and linguistic abilities. In order to better understand how all these behaviors interact, two techniques will be adopted for examining the agents’ internal (neural) representations. These analyses can identify and highlight the sensorimotor bases of linguistic structure, such as the role of perceptual and motor knowledge in differentiating the syntactic categories of verbs and nouns. The first methodology considers the phenomena of categorical perception in the hidden representations of neural networks. The second technique applies neuroimaging principles to artificial neural networks for comparing neural representations in different linguistic and behavioral tasks. The results of such analyzes will also be related to our current knowledge of the neural basis of language processing. 2. The neural processing of language and cognition The neuropsychological and neurocognitive literature on language processing in the brain is quite extensive (Pulvermuller, 2003). This has been possible through the use of a variety of scientific methods, such as neuropsychological experiments of patients with psycholinguistics deficits, brain imaging studies on normal adults and language-impaired patients and artificial neural network models. These methods support detailed investigations of the neural principles and mechanisms behind various linguistic abilities. For example, a recent review paper has analyzed numerous studies on the neural processing of the syntactic categories of nouns and verbs (Cappa & Perani, 2003). This review explicitly includes and compares brain imaging experiments on verb and noun processing (e.g. Martin et al., 1995; Perani et al., 1999). Cappa and Perani (2003) report an overall agreement on the fact that the left temporal neocortex plays a crucial role in lexical-semantic tasks related to the processing of nouns. The processing of verbs, instead, involves additional regions of the left dorsolateral prefrontal cortex. Such data show that language processing is strictly interconnected with other sensorimotor and cognitive abilities. For example, the prefrontal cortical areas implicated in verb processing are those also involved in motor control. This finding is consistent with other work that illustrates a strong relationship between language and motor development (Greenfield, 1991; Rizzolatti & Arbib, 1998). More generally, in cognitive psychology and neuropsychology it is widely accepted that language is not an autonomous function of the organisms, but it is strictly dependent on/from other cognitive abilities (Gazzaniga, 2000). In parallel to neuroscientific investigations, neural network models have extensively been used to study the neural control of linguistic behavior (Christiansen & Chater, 2001; Just et al. 1999). However, most of these models tend to study language in isolation. Only a few studies have directly and purposely simulated in the same model linguistic and motor functions (e.g. Reilly, in press). These include grounded adaptive agent models (e.g. Cangelosi 2001; Marocco et al. 2003), where by definition linguistic abilities emerge from sensorimotor and cognitive skills. Pre-Categorization After category learning After language learning Figure 1: Typical formation of clusters of points (i.e. square and circle categories) during category and language learning. Before category learning (Left), points corresponding to different categories overlap. After categorization (Centre) and language learning (Right) points group in distinct areas. Notice the enhancement of compression and separation (CP effects) between simple categorization and language learning. These diagrams do not correspond to real data, but only provide a sketch of CP effects observed in modeling experiments (e.g. Cangelosi et al. 2000). 2.1 Categorical perception Category learning has been hypothesized to be one of the core capabilities supporting the evolution and development of language and cognition. In particular, categorical perception (CP) is proposed as the basic mechanism sustaining our ability to build discrete and hierarchically-ordered representations of the environment (i.e. categories). Language and other higher-order cognitive abilities are then grounded upon such categories (Harnad, 1987; Cangelosi & Harnad 2000). CP refers to the effect of “warping” the similarity space of internal categorical representations. This results in the compression of within-category differences between members of the same category and the expansion of between-category distances amongst members of different categories. Such phenomena can be graphically represented through the process of the formation of clusters of points in the similarity space of categories (Figure 1). The first diagram (Fig. 1 Left) shows the undifferentiated similarity space before category acquisition. The other two diagrams represent the formation of quite distinct clusters (categories) after category learning has occurred without (Fig. 1 Centre) and with (Fig. 1 Right) language. These diagrams show the presence o