# A Framework for Real-Time 3D Freeform Manipulation of Facial Video
## 1 Introduction  
Social media has played a significant role on the way people can communicate and interact with others. With the advances in mobile devices and camera technology, activities such as taking and sharing photos or videos are increasingly a part of daily life. In particular, the \"selfie\" has become a dominant feature of online social media such as Facebook, Snapchat, or Instagram. This trend has been further boosted by filtering and editing capabilities, for example, making a face more beautiful or funny. As a new form of entertainment on social media, face editing (or manipulation) in photos or videos has been demonstrated by a variety of commercial tools such as the well-known professional software, Photoshop or mobile applications such as FaceApp [1], Face Warp [2], Face Warp 2 [3], Plastic Surgery Simulator [4], and Snow [5]. However, such solutions have a number of drawbacks from both a practical and technical point of view. For instance, Photoshop provides delicate functionalities for editing photos of human faces, but it takes time and effort for users to become familiar with these capabilities. FaceApp allows users to change facial appearance more easily, but fully automated operations often lead to undesired results. Face Warp provides some useful editing functions, but only for frontal faces. With Plastic Surgery Simulator, users can freely manipulate both facial and body photos, but it is difficult to make delicate or fine changes. Face Warp 2 and Snow have the advantage of being able to easily edit both photos and live videos of human faces, but the editable parts and range are limited to a few templates. Furthermore, most such face editing applications are based on 2D image processing, and so incur background distortion. For example, manipulating the face in Figure 0(a) using the existing applications can result in background distortion, as shown in the red circles of Figure 0(c),d. To address these performance limitations, we propose a new framework that allows 3 Dfreeform manipulation of a face in live video. Unlike existing approaches, the proposed framework fully employs 3D geometry of a face and provides improved results with real-time performance in video frames. Techniques for 3D editing of image and video have been diversely researched in the field of computer graphics [6; 7; 8; 9; 10; 11]. The main flow commonly involves:  
1. acquiring a 3D model of a target;
2. registering the 3D model and a 2D image of the target;
3. manipulating the model using 3D geometric transformation or deformation; and  
Figure 1: Results of face manipulation: (**a**) original image; (**b**) proposed framework; and comparison with existing applications: (**c**) Plastic Surgery Simulator; and (**d**) Snow. Red circles indicate image distortions incurred by manipulations. 4 texturing the manipulated model and rendering it on the image. Several approaches have been used to acquire a 3D model of a target, including computer vision-based 3D reconstruction from multiple images [6; 7], interactive 3D modeling from a single image under the assumption that its 3D shape is symmetric [8; 9], and obtaining from online 3D model database [10; 11]. Manipulation has also been implemented in various ways including copy and paste, geometric transformation, and isotropic or anisotropic scaling [8; 9; 10; 11]; deformation to an arbitrary shape [7; 8]; and physically based deformation [6; 11]. Even though such methods allow image or video editing based on 3D geometry of the target, they are unsuitable for face manipulation because the human face's range of expressive deformations cannot be modeled by a rigid body or a simple physical models as in the above methods. Face manipulation has been extensively studied and demonstrated on many different features such as transferring facial expressions to another face [12], replacing a face with another face [13], reshaping specific parts of a face [14; 15], and facial animation for a digital avatar [16; 17; 18; 19]. In some details relevant to our work, [PERSON] et al.