Discovering the Signatures of Joint Attention in Child-Caregiver Interaction

Discovering the Signatures of Joint Attention in Child-Caregiver Interaction Guido Pusiol Laura Soriano Li Fei-Fei Michael C. Frank guido@cs.stanford.edu lsoriano@stanford.edu feifeili@stanford.edu mcfrank@stanford.edu Department of Computer Science Department of Psychology Department of Computer Science Department of Psychology Department of Psychology Stanford University Stanford University Stanford University Stanford University Abstract Joint attention—when child and caregiver share attention to an object or location—is an important part of early language learning. Identifying when two people are in joint attention is an important practical question for analyzing large-scale video datasets; in addition, identifying reliable cues to joint atten- tion may provide insights into how children accomplish this feat. We use techniques from computer vision to identify fea- tures related to joint attention from both egocentric and fixed- camera videos of children and caregiver interacting with ob- jects. We find that the presence of caregivers’ faces in the child’s egocentric view and the motion of objects in the fixed camera both correlate with human-annotated joint attention. We use a classifier to predict joint attention using these fea- tures and find some initial success; in addition, classifier per- formance is substantially increased by interpolating features across automatically-extracted “attention chunks” in the ego- centric video. Keywords: Joint attention; computer vision; child develop- ment; social cognition. Introduction How do young children begin learning the meanings of words? Across cultures, early vocabulary includes names for people, simple social routines, animals, and objects (Tardif et al., 2008), suggesting that the earliest words are learned through interaction and play with others (Bruner, 1985). Identifying a caregiver’s intended referent is a critical part of learning meaning within these interactions, and this identifi- cation is often accomplished through joint attention. Joint attention describes the situation when both child and caregiver are attending to the same thing and when both know that the other is attending to it. For the remainder of the paper we will talk informally about joint attention—JA—as both the phenomenon and the period of time during which it hap- pens (Carpenter & Liebal, 2011). A typical example of JA is a situation where an adult and child are playing with a toy and the infant alternates gaze between the adult and the toy (Carpenter, Nagell, & Tomasello, 1998). The capacity for JA gradually develops over the first two years of life and usually begins to emerge between 9 and 12 months of age, coinciding with the beginnings of lan- guage learning. In addition, both the skills that enable JA (e.g. pointing, following a caregiver’s gaze to a distal target) and the amount of time that children spend in JA with their caregivers are strong predictors of children’s early vocabulary growth (Carpenter et al., 1998; Brooks & Meltzoff, 2008). How do children know that they are in joint attention with a caregiver? From an external perspective, joint attention has typically been defined by a sequence of events: (1) one mem- ber of the interaction (child or caregiver) directs the other members attention to an object, (2) both members focus vi- sually on the object, and (3) the child indicates awareness of the caregiver (Tomasello & Farrar, 1986). Previous work has used children’s gaze as the main indi- cator of JA, but, from the perspective of both the child and the data analyst, this method has several issues. First, gaze is neither necessary nor sufficient for JA. It is possible to attend jointly through the hands—as with a child reading a picture- book on a parent’s lap—or for the child to follow gaze to a distal target and then signal awareness by moving towards it or reaching for it (Yu & Smith, 2013). Indeed, eye-tracking studies investigating signals to reference find that manual sig- nals are far more effective than gaze in manipulating young children’s attention (Yurovsky, Wade, & Frank, 2013). Sec- ond, young children may not have perceptual access to their caregiver’s gaze most of the time. Recent studies using head- mounted cameras and eye-trackers suggest that children are more often looking at the objects in front of them than at the faces of their caregivers (L. B. Smith, Yu, & Pereira, 2011; Franchak, Kretch, Soska, & Adolph, 2011; Frank, Simmons, Yurovsky, & Pusiol, 2013). Third, parents most often look at their children, not at the object they are talking about (Frank, Tenenbaum, & Fernald, 2013). Thus, gaze alone is at best a noisy cue for the identification of JA, either for the child or for the researcher attempting to identify JA in a large dataset. The goal of our current work is to discover other signals of joint attention. There are two purposes to this investigation. The first is data analytic: A better understanding of how to extract JA episodes from video could be a powerful tool for analyzing large video corpora. The second is psychological: The unsupervised extraction of JA episodes from video could give hints regarding robust cues that children might use in addition to, or even in lieu of, gaze. We use two data sources to gain information about the so- cial interaction between child and caregiver: head-mounted and fixed camera videos. Our approach is unsupervised dis- covery. We hypothesized that the most effective strategy for capturing JA would be the extraction of high-level, seman- tic features that correspond relatively closely to the kinds of constructs described in prior work manually coding joint at- tention (e.g. Tomasello & Farrar, 1986). Of course, the chal- lenge is that many such features can be extremely difficult to extract in an automated fashion. To compromise, we identi- fied three features that we could extract with relatively high accuracy in an automated fashion: (1) caregivers’ faces in the egocentric camera, (2) objects that were in motion due to being actively manipulated, and (3) periods of time during