Grounding word learning in multimodal sensorimotor interaction

Grounding Word Learning in Multimodal Sensorimotor Interaction Chen Yu, Linda B. Smith and Alfredo F. Pereira (chenyu@indiana.edu) Department of Psychological and Brain Sciences, and Cognitive Science Program, Indiana University Bloomington, IN, 47405 USA Abstract A central problem in the study of language acquisition is word learning – how the child’s mental representation of objects and events on the one hand is associated with the linguistic input on the other, and how young children acquire the vocabulary of their first language so effortlessly and smoothly, with virtually no errors along the way. This paper aims at understanding the mechanisms through which word learning is grounded in sensorimotor experience, in the physical regularities of the world, and in the time-locked and coupled multimodal interactions between the child’s own actions and the actions of their caregivers. We designed and implemented a novel multimodal sensing environment consisting of two head-mounted mini cameras that are placed on both the child’s and the parent’s foreheads, motion tracking of head movements and recording of caregiver’s speech. Using this new technology, we captured the dynamic visual information from both the learner’s perspective and the parent’s viewpoint while they were engaged in a naturalistic toy-naming interaction, to study the regularities and dynamic structure in the multimodal learning environment. To achieve this goal, we also implemented various data processing programs that can automatically extract visual, motion and speech information from raw sensory data. Our results show that a wide range of perceptual and motor patterns, such as the proportion of the named objects in both the child’s and the caregiver’s visual fields, the proportion of time that the child’s hands are holding the named objects when those names are uttered, and as well as the child’s head movements, are predictive of successful word learning through social interaction. In light of this, we suggest that high-level social- cognitive cues in word learning can be grounded in embodied perceptual and motor patterns that are part of a natural social interaction. Keywords: language development, embodied cognition, learning, computational modeling Introduction A major recent advance in understanding word learning has been the documentation of the powerful role of social- interactional cues in guiding infants’ attention and in linking the linguistic stream to objects and events in the world (Baldwin, 1993; Tomasello & Akhtar, 1995; Yu, Ballard & Aslin, 2005). There can be no doubt that young learners are highly sensitive to the social information in these interactions (e.g., Baldwin, 1993; Bloom, 2000; Woodward, 2004). However, the nature of this sensitivity and the relevant underlying processes are far from clear. Often in this literature, children’s use of social cues is interpreted in terms of (and seen as diagnostic markers of) their ability to infer the intentions of the speaker. This kind of social cognition is called “mind reading” by Baron-Cohen (1995). Butterworth (1991) showed that even by 6 months of age, infants are sensitive to social cues, such as monitoring and following another’s gaze, although infants’ understanding of the implications of gaze or pointing does not emerge until approximately 12 months of age. Based on this evidence, some researchers (e.g. Bloom, 2000, Tomasello, 2000; Woodward, 2004) have suggested that children’s word learning in the second year of life actually draws extensively on their understanding of the thoughts of speakers. However, there is an alternative explanation to that of “mind-reading”. Smith (2000) has suggested that these results may be understood in terms of the child’s learning of correlations among actions, gestures and words of the mature speaker as predictors of attention and intended referents. Smith (2000) argued that construing the problem in this way does not “explain away” notions of “mind- reading” but rather grounds those notions in the perceptual cues available in the real-time task that infants must solve (see also Smith & Breazeal, 2007). One problem with resolving (or integrating) these two ideas about the information available in social interactions is that most experiments are designed as demonstrations of children’s sensitivity to social cues and as such, they focus on macro-level behaviors (such as pointing, or direction of eye gaze) in constrained contexts in which the experimenter presents one or two objects on an uncluttered table and where language is also uncluttered by remarks about anything other than those objects on the table (e.g. Baldwin, 1993). To truly understand mechanisms of learning, however, we may need to focus on more micro-level behaviors as they unfold in real time in the richly varying and dynamically complex interactions of children and their mature partners in more naturalistic tasks (such as toy play). Further, whereas the studies at the macro-level clearly demonstrate many intelligent behaviors in infant word learning, they have not as yet led to a formal account of the underlying mechanisms. Thus, we want to know not only that learners use social cues but also how they do so in terms of the real-time processes in the naturalistic tasks where everyday language learning must take place. To this end, we sought to study the dynamics of social cues to word learning at the sensorimotor levels – in the bodily gestures and as well as momentary visual and auditory perception of the participants. The study presents a new design and implementation of a sensing system for recording multisensory data from both the child and the caregiver. With this new methodology, we compare and analyze the dynamic structure of natural parent-child interaction in the context of language learning, and further discover perceptual and motor patterns that are informatively time-locked to words and their intended referents and predictive of word learning.