Title Facial expression modeling using component AAM models-Gaming applications

In this paper we provide preliminary results on a new modeling approach for improved determination of facial expressions from a low-resolution video stream. An initial proofof-concept using an extension of an Active Appearance Model (AAM) to measure facial parameters and a set of classifiers to determine facial states are described. We also discuss the potential for applying this technique to determine the emotional state of the players of a computer game and suggest how this information can be integrated into the workflow of a game. In addition we describe a number of use cases where the game environment can be adapted based on feedback from the players. Keywordsinteractive gaming, affective imaging, expression classification and recognition, AAM, SVM I. INTRODUCTION (HEADING 1) Computer gaming has grown from its humble origins to become a global industry rivaling the movie industry in terms of scale and economic impact. The technology of gaming continues to improve and evolve at a very rapid pace both in terms of control interface and the graphical display of the gaming world. Today's user interfaces feature more sophisticated techniques for players to interact and play cooperatively with one another. It is possible to have real-time video and audio links between the real players so they can coordinate their group gameplay. However the emphasis remains on the player being drawn into the artificial game world of the computer. There is still little scope for the gaming environment to reach back to the the players, sensing and empathizing with their moods and feelings. Given the sophistication of modern AI game engines we feel this is a missed opportunity and that gaming engines will soon need to evolve to develop and provide methods to empathize with individual game players. In this paper we present one practical approach to providing a direct connection back to the game player. We describe an approach to modeling and classifying the facial expressions of a game player using a low-resolution webcam and state-of-art face detection and face modeling techniques. Our system in unoptimized form, is capable of running in real-time on a standard desktop PC. With further refinement it could be implemented on a dedicated embedded device, or converted to a dedicated hardware subsystem. However, in this paper we are primarily concerned with presenting a working proof-ofconcept. This paper is organized as follows. An overview of our system is given in section II; section III describes how facial features can be extracted using AAM models; the section IV and V respectively describe improved eye and lips models; section VI describe the full component model for a face region and section VII explains how we optimized the relevant features which are extracted using this model. In Section VIII we explain our approach to train classifiers and how these are used to determine facial expressions. In section IX a range of representative results are presented showing the discriminative abilities of our system between different facial expressions. Finally in section X we discuss some applications of our system in computer gaming, draw some conclusions and present proposals for future work. II. SYSTEM ARCHITECTURE A system that performs automatic face recognition or expression recognition typically is comprised of three main subsystems, as shown in Figure 1: (i) a face detection module, (ii) a feature extraction module, and (iii) a classification module which determines a similarity between the set of extracted features and a library of reference feature sets. Other filters or data preprocessing modules can be used between these main modules to improve the detection, feature extraction or classification results. Figure 1: Generic Facial Analysis/Classification System In the first module, it is decided whether the input picture or the input video sequence contains one or more faces. If so, then facial features are extracted from the detected faces, in our case by applying an advanced face model which encodes the facial features by a set of parameters. As a third step, the facial features  determined as a set of output parameters from the model are classified in order to perform facial recognition or expression classification. A. Face Detection Module In our system, we employ a face detector module as initialization for the AAM search. Face detection can be defined as the ability to detect and localize faces within an image or a scene. In the last few years, many different face detection techniques have been proposed in literature. A comprehensive survey of the face detection methods is presented in [1]. State-of-the-art face detection methods provide real-time solutions that report high detection rates. In this field the most significant advances in the last decade are due to the work of Paul Viola and David Jones who proposed a face detector based on rectangular Haar classifiers and the integral image representation of an input image [2]. This detector is the fastest face detector reported in the literature so far. It is able to perform for semi-frontal faces in real-time and is highly accurate. The Viola-Jones face detector is presently the gold standard against which other face detection techniques are benchmarked. In our system, we employ the Viola-Jones face detector, as initialization for the AAM search. An example of face detection using the Viola and Jones algorithm is shown in Figure 2. This algorithm has been implemented in OpenCV [3], [4] which is a free computer vision library used widely by the computer vision research community. Figure2: Examples of detected faces using the Viola-Jones algorithm. B. Feature Extraction Module In our work, we extend the AAM approach for facial feature extraction. Although AAM is a powerful tool for image interpretation, the conventional algorithm is unreliable when dealing with generalized facial expressions. It is a global face model whereas the key features which determine facial expressions are local features. It is these local features which are responsible for most of the relevant facial variation. Component-based AAM [5] offers a practical solution to these drawbacks. It combines a global face model with a series of sub-models. These sub-models are typically component parts of the object to be modeled. This approach benefits from both the generality of a global AAM model and the local optimizations provided by its sub-models. We adjust the model through training to be robust to small to medium pose variations and to directional illumination changes. We then demonstrate the benefits of our system, with respect to expression classification and recognition, taking into account the real-time requirements of such a system. C. Expression Classification Module In our work, two classifiers are compared: Nearest Neighbor (NN) and Support Vector Machine (SVM). The choice of the two classifiers is based on their positive results obtained in the literature. In [6] it is stated and it is proved by experiments for gender classification and face recognition that SVM is typically among the top two classifiers, and the other top ranking classifier is one of the Euclidean-NN rule or the cosine-NN rule [7]. As we designed them, in this work these classifiers use the relevant AAM parameters to choose between facial expressions. III. THE EXTRACTION OF FACIAL FEATURES USING AAM A. Statistical models of appearance (AAM) AAM was proposed by Cootes et al. [8] in 1998 as a deformable model, capable of interpreting and synthesizing new images of the object of interest. The desired shape to be modeled  in our case a facial region is annotated by a number of landmark points. A shape vector is given by the concatenated coordinates of all landmark points and may be formally written as, s=(x1,x2, ,xL,y1,y2, ,yL), where L is the number of landmark points. The shape model is obtained by applying Principal Component Analysis (PCA) on the set of aligned shapes: