Intelligent facial animation: Creating emphatic characters with stimuli based animation

The purpose of this PhD research is to define a novel method for real-time, stimuli based, procedural facial animation with cinematographic quality. In procedural methodologies the artist does not control the animation directly, only provides the initial parameters and defines a set of rules, which are used to generate the animation. Procedural animation has been used for body animation, while the face remains an unexplored area, where traditional techniques prevail. Conventional facial animation methods include: key frame animation, which consists on having an artist create all the (facial) poses by hand, therefore becoming an extremely time consuming task and performance-based techniques that can speed up the animation process but still require cleaning the motion data manually. Thus, generating facial animation automatically is a demand but also a challenge, since it is very difficult to predict all possible facial movements, becoming even harder if the character is supposed to react to external stimuli. The key contributions of this research will be: a procedural animation method, an intelligent agent that will automatically react to external stimuli, a full study of the reactions caused by these stimuli and how they affect the face. As a result there will be a reduction of the time required to create a facial animation. It will have great impact in both films industry and immersive systems such as videogames and medical systems, by reducing the time required to animate a character and creating interactive, highly empathic human-like characters. Detailed Description: In this thesis the main research challenges to overcome are: which are the triggers to generate a specific expression, how this expression is created and how to automatically do generate animation. Currently there is no standard to how an external event triggers an action in real life applied to facial animation. One possible case could be to pinch someone on the face. What would the facial expression of that person look like? Would he/she cry? Become angry? Both? This depends not only on personal factors, such as age and emotional state, but also the context of the event. Was it a friend? Were they fighting? Or maybe playing a game? Also, how do the context, personal factors and triggers contribute to creating the pose? Although the deformations that occur on the model, as a result of a direct interaction (i.e. the pinch), have already been properly explored, the afterwards reaction has not. This is due to the complexity of the task and lack of methods for easy animation of the aftermath. The overall goal is to create a system capable of reacting to external stimuli and generate proper facial animations in an automatic way, thus creating a procedural facial animation method. To achieve this goal the research plan is divided into three stages: S1: To study and understand how different stimuli lead to the creation of different poses and create a methodology based on how these stimuli affect facial movements; S2: To create a procedural method with S1 as a basis capable of generating, with few parameters, any expression. S3: To research intelligent agents capable of using the data of the studies performed in S1, to simulate human-like reactions and combine the agent with the results of S2 S4: To validate the whole system. S1: Creation of a methodology to represent facial movements: The methodology will be based on voluntary and involuntary movements. To create the rules that define the methodology it is necessary to understand how psychological and physiological reactions to a stimulus affect the face. A similar study has already been made for body animation [1], but as far as it is known none exists for the face. Research will be done by analysing how the poses are created from psychological, psychiatric and neurological points of view. Regarding voluntary movements, they are the result of the cognitive functions of the brain e.g. speech or an expression. Involuntary movements are the result of the autonomous areas of the brain, which result of an illness or a psychological factor such as a nervous tic. To better understand how to create the set of rules, the FACS [2] system will be analysed. This system can be used to animate a face even though it was developed from a psychological perspective. It defines a set of action units (basic element), which in turn are associated to sets of muscles. This system limits the creation of different poses and does not have the notion of motion as procedural animation requires. Facial movement will also be analysed from a psychological point of view. The goals of this step are: S1.1: To study trigger-reaction causality pairs, initially for controlled cases and then generalize the study; S1.2: Based on S1.1, identify involuntary and voluntary movements, caused by different reactions; S1.3: To define a set of rules for facial movements, based on S1.2, thus creating the facial animation methodology. S2: Create the facial procedural animation method: The method will be based on S1.3. It will allow high-level commands based on the defined rules and will be the interface between the animator and the method. To achieve this method the current state of both, facial animation and body procedure animation will be studied and used as a layer to implement the rules (e.g. use a muscle system and on top of it create the rule system). Using this method, it will be possible to create and change facial animations. The goals are: S2.1: To study full body procedural animation S2.2: To create a method for procedural facial animation based on S1.3 and S2.1 S3: Create an intelligent agent that is capable of reacting to external stimuli. After creating the animation procedure, a layer of behaviour will be added with the purpose of creating a completely autonomous agent. Artificial intelligence methods (for instance reinforcement learning methods) will be studied and used as a basis for creating the agent. By adding an intelligent agent to the procedural method, it is possible to create a fully automatic animation agent, where the animator’s role is replaced by the agent’s. The agent is capable of reacting to external stimuli, passing the action to the animation procedure, which will allow creating animations on the fly. This agent will be used to automatically animate the talking head producing a human-like behaviour. The main goals are: S3.1: To study artificial intelligent agents S3.2: To create an intelligent head and apply to it the procedural facial animation method. This step is based on S2.2 and S3.1 S4: Validation. The final step is to validate the whole system through qualitative and quantitative methods. The qualitative validation will be based on empirical methods (observation and enquiries) and will be made for both artists and viewers. The tests will be the visual quality of both: a comparison between traditional animation and the system’s quality and ability of the character to convey emotions based on subtle expression. The quantitative tests will also be based on the same comparison and will include: the time required to create animations, the reaction time of the character and a comparison between the muscles moved by the system and the ones a person would move in a similar situation amongst others. The goals are: S4.1: To validate the system in a qualitative and quantitative way