# A Framework for Real-Time 3D Freeform Manipulation of Facial Video
## 2 Facial Manipulation Framework
### Registration  
The registration process involves two steps: facial landmark detection and face model fitting. For facial landmark detection, feature points (or interest points) around the eyes, nose, mouth,  
Figure 2: Workflow of the proposed face manipulation framework. and facial outline are detected in a video frame. These points are then used as reference ones for fitting a face model on a facial region in the frame. Among the various methods of facial landmark detection, in recent years, random regression forest [24; 25] and deep learning [20] algorithms have actively been used. Random regression forest-based methods are super-fast, yet highly dependent on initial guess, causing to erroneous detection. Deep learning-based methods provide robust detection, whereas the computational cost is high. One of main concerns in this study was accurate and robust facial landmark detection under various head poses and face shapes because registration errors are easily noticeable, so that the manipulation looks unnatural; in the proposed framework, we therefore take advantage of the deep learning-based method, proposed by [PERSON] et al. [20]. Basically, this method builds a multistage deep neural network with several serially connected convolutional neural networks. At each stage, geometric transformation between input frame and the canonical shape of the face, the landmark heat map, and the feature map are estimated, and the locations of facial landmarks are refined by using them. Based on the detected facial landmarks, a face model is fitted on a face region in the frame by updating and refining their corresponding landmarks on the model and head poses in an iterative manner. However, the face model fitting is quite challenging due to various facial shapes and expressions. To deal with them, the proposed framework adopts a 3 DMM [21], which can represent various shapes and expressions, and further improves the face model fitting to allow real-time performance. Conventionally, the 3 DMM is generated from a training set of 3D face scans and represented using principal component analysis (PCA) as a linear combination of basis vectors:  
\[\mathbf{M}=\mathbf{\tilde{M}}+\mathbf{M}_{sb}\cdot\boldsymbol{\alpha}+\mathbf{ M}_{cb}\cdot\boldsymbol{\beta}, \tag{1}\]  
where \(\mathbf{\tilde{M}}\) is the mean face model with neutral expression; \(\mathbf{M}_{sb}\) and \(\boldsymbol{\alpha}\) are the shape basis and coefficients; and \(\mathbf{M}_{cb}\) and \(\boldsymbol{\beta}\) are the expression basis and coefficients. Here, \(\mathbf{M}\) is the 3D facial mesh, defined by a set of 3D vertices \(\mathbf{X}\), \(\mathbf{M}=[\mathbf{X}_{1},\mathbf{X}_{2},...,\mathbf{X}_{n}]^{T}\in\mathbb{ R}^{3}\). Since the facial landmarks are annotated on the mean face model in advance, head pose can initially be estimated using the landmarks and their correspondences on the model. With the initial head pose, the shape coefficients can be computed by minimizing the following cost function, written by the reprojection error of the landmarks:  
\[\bar{\boldsymbol{\alpha}}=\underset{\boldsymbol{\alpha}}{\arg min}\sum_{i=1}^{N} \frac{\left(\mathbf{P}\mathbf{X}_{i}\left(\boldsymbol{\alpha}\right)-\mathbf{ x}_{i}\right)^{2}}{2\sigma^{2}}+\lambda\left\|\boldsymbol{\alpha}\right\|_{2}^{2}, \tag{2}\]  
where \(N\) is the number of landmarks, \(\sigma\) is the standard deviation of landmarks, \(\mathbf{P}\) is the camera projection matrix, \(\mathbf{x}_{i}\) and \(\mathbf{X}_{i}\left(\boldsymbol{\alpha}\right)\) are the \(i\)th facial landmark (2D position) detected from the video frame and the corresponding landmark (3D position) on the model, and \(\lambda\) is the regularization parameter.