DOI: 10.25549/usctheses-c40-342951
title: Analysis and modeling of multi-level dynamics of multimodal behavior in affective human interactions

description: Human communication is a dynamical and interactive process that is established on a common ground of conveying emotions, achieving the interaction goals and sharing mutual interests of the interaction participants. Such an interactive process naturally induces a multi-level dynamical flow along various verbal and nonverbal behavior dimensions of spoken words, speech prosody, body gestures, and facial expressions. As one of the major components that shape the structure of social interactions, emotions greatly affect the multi-level dynamics of multimodal behavior throughout the course of an interaction. This thesis, from multiple perspectives, explores computational methodologies to understand, analyze and model human behaviors dynamics that relate to and arise from affective processes underlying human interactions. ❧ From the first perspective of understanding self-evolution dynamical traits of a single modality behavior, we propose a statistical framework for robustly modeling the dynamics of body gesture expression over an affective interaction. We focus on modeling the high degree of gesture variability across individuals and over different time scales. The framework is designed based on the notion of gesture patterns that define the basic gesture elements. Experimental results show the superiority of the proposed approach in emotion recognition tasks. The analysis of model parameters sheds light on the understanding of the underlying structure of human summarization process of the global emotion perception. ❧ This thesis further studies the interrelationship across modality behavior (speech and gesture) of an individual over the course of an interaction. Analysis results quantitatively show that both strength and structure of speech-gesture coupling are controlled and modulated by the underlying emotional content. Extending the quantitative analysis on speech-gesture interplay, we focus on lightly-supervised emotion recognition by modeling the relationship among multimodal behavior. We describe a latent emotion modeling approach using canonical correlation analysis (CCA) based multimodal words to probabilistically infer emotion clusters from multimodal behavior data with only a small number of emotion labels. This work brings insights into large-scale emotion learning in an ecological manner. ❧ Finally, this thesis also contributes to the aspect of interpersonal behavior coordination in a dyadic interaction. We computationally explore how body language behavior is coordinated across interaction participants to express emotions. We experimentally demonstrate the existence of interpersonal coordination and its dependence on the emotional states. Motivated by the analysis results, we further focus on interaction modeling for improving emotion recognition performance. The interpersonal behavior coordination is parameterized through a geodesic flow that connects the behavior subspaces of two interaction participants. Such geodesic flow based modeling exploits both geometric and statistical properties of a dyad’s behavior, and mathematically describes their interaction structure.