Affect Analysis using Multimodal Cues

In this PhD project, I am working on the problem of multimodal affective sensing with focus on developing an objective measure for diagnosing and monitoring depression using multimodal cues, such as facial expressions, body gestures or vocal expressions. As the depression severity of a subject increases, the facial movements become very subtle. In order to quantify depression and its subtypes, we need to reveal these changes. A particular focus of my research is on improving the ability of affective computing approaches, to sense the subtle expressions of affect in face and voice, and to develop approaches that measure the severity of depression. In my work till date, I have investigated the sensitivity and specificity of affective computing approaches and demonstrated that these methods can successfully distinguish subtypes of depression. These results will serve as a baseline for future development towards a more fine grained depression classification and analysis. I. PROBLEM AND MOTIVATION This highly inter-disciplinary PhD project will investigate the fundamental affective computing problem of robust noninvasive multimodal approaches for sensing a person’s affective state, with a particular focus on measuring the affect intensity. The affect intensity measure gives an indication of how strongly or weakly individuals tend to experience emotions in their everyday life. Affective computing [1] relates to adding emotional intelligence to computers such that the system is capable of recognizing, analysing and synthesizing human emotions. It expands Human-Computer Interaction (HCI) by including affective (emotional) communication together with appropriate means of handling affective information. In this project, multimodal affective sensing approaches will be investigated and developed in general. Novel paradigms to address current open research questions in affective sensing will be explored. The utility of such approaches will be validated on commonly used affective computing benchmark problems, such as the realworld example of quantifying depression severity. Earlier work has shown that current affective sensing approaches can successfully distinguish distinct affect classes. For example, Joshi et al. [2] showed that a multimodal approach can correctly classify healthy controls and severely depressed subjects. However, current affective sensing approaches fail when the expressions of affect are very subtle. The aim of this project is to address the much more difficult problem of quantifying affect intensity on a continuous range as well as dealing with heterogeneous classes such as major depression [3], where the usual categorical approach such as depressed vs. non-depressed does not work, as often found in real-world applications of affective sensing technology [4]. I will exemplify my investigations on a number of commonly used affective sensing datasets [5], [6] around the benchmark problem of quantifying depression and melancholia. A multimodal system to objectively assess the severity of depression would provide a major breakthrough in mental health care and have significant potential for further research and commercialisation.