Segmenting and Recognizing Human Action using Low-level Video Features

Segmenting and Recognizing Human Action using Low-level Video Features Daphna Buchsbaum, Kevin R. Canini, Thomas L. Griffiths daphnab@berkeley.edu, kevin@cs.berkeley.edu, tom_griffiths@berkeley.edu University of California, Berkeley, Berkeley, CA 94720 USA Abstract Dividing observed human behavior into individual, mean- ingful actions is a critical task for both human learners and computer vision systems. An important question is how much action structure and segmentation information is available in the observed surface level motion and image changes, without any knowledge of human pose or behavior. Here we present a novel approach to jointly segmenting and recognizing videos of human action sequences, using a hierarchical topic model. Video sequences are represented as bags of video words, automatically discovered from local space-time interest points. Our model jointly infers both action identification and action segmentation. Our results are a good fit with human segmentation judgments as well as providing relatively accurate action recognition and localization within the videos. Introduction When observing a continuous stream of human behavior, an ongoing problem for both human learners and computer vision algorithms is recognizing and segmenting out indi- vidual, meaningful actions from within the longer motion sequence. This ability to pick out individual actions or events is thought to play an important role in human social cognition (Baldwin & Baird, 2001), allowing us to recognize other’s behavior, and helping us link people’s actions with their internal mental states, and with the outcomes of those actions (Buchsbaum, Griffiths, Gopnik, & Baldwin, 2009; Zacks, Speer, Swallow, & Maley, 2010). It is also a critical first step in developing computational approaches that can recognize and understand human action from naturalistic videos. Prior research has shown that adults are able to segment videos of common everyday activities into coherent actions, consistent with the goals and intentions underlying the actor’s behavior e.g. (Newtson, Engquist, & Bois, 1977; Zacks, Tversky, & Iyer, 2001), and that even young infants are sensitive to the boundaries between intentional action segments (Baldwin, Baird, Saylor, & Clark, 2001; Saylor, Baldwin, Baird, & LaBounty, 2007). People’s boundary judgments are also sensitive to the hierarchical structure of goals and sub-goals underlying human action – they are able to consistently segment actions at multiple levels of granularity (Zacks, Tversky, & Iyer, 2001; Zacks, Braver, et al., 2001; Hard, Tversky, & Lang, 2006). While a full understanding of human action requires knowledge about goals and intentions, infants are able to parse dynamic human action well before they are thought to have a fully developed theory of mind. This suggests that there may also be low-level cues to intentional action structure available in human motion, an idea supported by a variety of recent work (Zacks, 2004; Hard et al., 2006; Zacks, Kumar, Abrams, & Mehta, 2009; Meyer, DeCamp, Hard, & Baldwin, 2010; Hard, Recchia, & Tversky, under review). In particular, movement features within simple animated scenes (Zacks, 2004), as well as movement and position of tracked body parts within videos of everyday action (Zacks et al., 2009; Meyer et al., 2010), have been shown to correlate with human segmentation judgments. However, this previous work still assumed some a priori knowledge of human body pose, and of the types of motion features relevant to boundary detection. In this paper, we present a computational model that makes very few representational assumptions about what is observed, in order to explore the amount of action structure that can be inferred from just low-level changes in pixel val- ues, without knowledge of human body structure, higher level goals and intentions, or even foreground/background distinc- tions. To the extent that the model corresponds to human segmentation judgments, and correctly recognizes actions, we know that there are cues in surface level image changes that can be used to both segment and identify human behavior. In addition, the problem of determining which subse- quences of behavior go together can be seen as an important instance of the more general problem of variable (or feature) discovery and segmentation, a problem that the psycho- logical, machine learning and computer vision literatures have addressed in other domains. Recent work in action segmentation has drawn particularly on both psychological and computational approaches to the problem of segmenting words from continuous speech. There is now a large body of evidence suggesting that both infants and adults can use statistical patterns in spoken language to help segment speech into words (for a partial review, see Gomez & Gerken, 2000). More recently, it was demonstrated that a similar sensitivity to statistical regularities in action sequences could play an important role in action segmentation e.g. (Baldwin, An- dersson, Saffran, & Meyer, 2008; Buchsbaum et al., 2009). Similarly, Buchsbaum et al. (2009) were able to successfully adapt a Bayesian model of statistical word segmentation (Goldwater, Griffiths, & Johnson, 2009) to the action domain. However, like previous computational models of word segmentation, the Buchsbaum et al. (2009) model assumes that the lowest level of segmentation is already known (or pre-labeled). That is, that there is some sort of motion primitive (equivalent to a syllable or phoneme in speech), that can already be recognized as a coherent unit. Since psy- chological studies demonstrating human action segmentation have suggested that statistical patterns or features in human motion may correlate with segment boundaries at even the lowest level, we would like to see whether action boundaries can be automatically detected directly from video, without pre-existing knowledge of low-level motion units. Finally, in the computer vision literature, most action recognition work has focused on pre-segmented videos of