# A Framework for Real-Time 3D Freeform Manipulation of Facial Video
## 2 Facial Manipulation Framework
### Manipulation  
After the face model has been fitted on the video frame, it can be deformed by the user's input, based on the ARAP method [22]. ARAP is a mesh deformation method that allows natural deformation while maintaining the rigidity of the local surface through a two-step optimization with enforced rotational transformation. Given a mesh model \(\mathbf{M}\) with \(n\) vertices, deformation energy \(E\) can be formulated in terms of its vertices \(\mathbf{X}_{i},\ i\in\{1,...,n\}\) and their neighbors \(\mathbf{X}_{j}\in N\left(\mathbf{X}_{i}\right)\) as  
\[E=\sum_{\mathbf{X}_{i}\in\mathbf{M}}\sum_{\mathbf{X}_{j}\in N\left(\mathbf{X} _{i}\right)}\omega_{ij}\left\|\left(\mathbf{X}_{i}^{\prime}-\mathbf{X}_{j}^{ \prime}\right)-\mathbf{R}_{i}\left(\mathbf{X}_{i}-\mathbf{X}_{j}\right)\right\| ^{2}, \tag{4}\]  
where \(\omega_{ij}\) is the weight, \(\mathbf{X}_{i}^{\prime}\) and \(\mathbf{X}_{j}^{\prime}\) are the positions of \(\mathbf{X}_{i}\) and \(\mathbf{X}_{j}\) after the deformation, and \(\mathbf{R}_{i}\) is the rotation matrix of the local surface defined by \(\mathbf{X}_{i}\) and \(N\left(\mathbf{X}_{i}\right)\). In the two-step optimization, \(\mathbf{X}_{i}^{\prime}\) and \(\mathbf{X}_{j}^{\prime}\) are unknown variables to be estimated, playing different roles as three types of anchor constrained by a user:  
* static anchors (vertices to be fixed);
* handle anchors (vertices to be moved by a user); and
* ROI anchors (vertices of the region of interest, to which the deformation is propagated).  
In the first step of optimization, the rotation matrices for the local surfaces are computed with the positions of ROI anchors being fixed. Then, the rotation matrices are fixed, and the positions of the moving vertices are calculated by minimizing the cost function. Figure 4 shows our results for mesh deformation by a user's input. On the original mesh model (Figure 4a), handle anchors and ROI anchors were set by the user (Figure 4b), and the model could instantly be deformed when the positions of the handle anchors were changed by the user (Figure 4c,d).  
Basically, an anchor is taken by picking a vertex. When a user specifies 2D screen coordinates by clicking a mouse, the 2D coordinates are backprojected and converted to a ray. Then, a facet is selected by computing the intersection between the ray and the model, and the closest vertex which belongs to the facet will be the anchor. In other words, when a spot on a face model is chosen by a user, its closest vertex of the model is defined as an anchor. Although every vertex on the model can be the handle or the ROI anchors, it can lead to discontinuities in the borders with areas that the face model doesn't cover.  
On the other hand, the face model fitted in the registration module is not suitable for selecting and moving the handle anchors because it is changed for every frame according to face viewpoint and expression. For that reason, the proposed framework performs the deformation on the mean face model in the neutral state rather than directly using the fitted face model. In this regard, the face model in Equation (1) is rewritten by an additional term for the final rendering:  
\[\mathbf{M}^{\prime}=\left(\bar{\mathbf{M}}+\mathbf{M}_{sb}\cdot\bar{\mathbf{ \alpha}}+\mathbf{M}_{eb}\cdot\bar{\mathbf{\beta}}\right)+\mathbf{W}\left(\mathbf{ M}_{m}-\bar{\mathbf{M}}\right), \tag{5}\]  
where \(\mathbf{M}_{m}\) is the deformed model in the manipulation module and \(\mathbf{W}\) is the weight matrix that controls the degree of deformation of each vertex. Here, the weights for every vertices in the \(\mathbf{W}\) are set to 1 in all cases, and the user deformation is applied as is--that is, the model is determined by merging the fitted model with the difference between after and before mesh deformation.