# A Framework for Real-Time 3D Freeform Manipulation of Facial Video
## 3 Experimental Results
### Performance Evaluation and Comparison  
To verify real-time performance, we evaluated the processing time of the proposed framework on a desktop PC (Intel i7-3770 CPU and NVidia GeForce GTX TITAN GPU) using a face model with 3448 vertices and 6736 triangles and a video with resolution of 856 \(\times\) 480. At first, for the frontal and oblique face, processing times in face model fitting were computed with and without two acceleration schemes: parallel processing with OpenMP and correspondence filtering (as described in the last paragraph of Section 2.1). As shown in Table 1, processing time was sped up in every case, regardless of head pose. For the oblique face in particular, processing time was drastically dropped by about 37 ms with OpenMP and 29 ms with correspondence filtering. Note that, since neither acceleration affected whole face model fitting process, processing times were not proportional,  
Figure 7: An example of augmented reality application using the proposed framework.  
regardless of acceleration applied. Even though face model fitting functioned in real time with acceleration, the overall performance of the framework might suffer from other heavy tasks such as landmark detection, mesh deformation, and inpainting. As shown in Table 2, total processing time was about 113 ms per frame (9 fps) on the assumption that entire processes were sequentially performed, which might not be sufficient for real-time applications. Moreover, processing time may be further increased because mesh deformation can run several times within one frame when the user is moving handle anchors. In the proposed framework, however, inter-module parallelization scheme enabled carrying out the entire processes at about 15 fps, derived from the least upper bound (58.49 ms) of subtotal processing times.  
For qualitative evaluation, we compared our manipulation results with two existing applications: Plastic Surgery Simulator [4] (Figure 8) and Snow [5] (Figure 9). In the former case, it was difficult to manipulate the face precisely and consistently because the deformation was performed manually for each frame by stretching or crushing given point of the 2D frame and its surrounding area; thus, as shown in Figure 8c, artifacts were incurred in several deformed parts (e.g., stretched nose), and background regions were distorted around the neck or t-shirt. In Snow, manipulation was often accompanied by background distortion. Natural manipulations were possible only for the frontal face and became worse with head pose changes (e.g., turning the face sideways or leaning forward), as shown in Figure 9a. Moreover, only a few predefined deformations could be selected and applied. Different from them, our results show that natural 3D face manipulation could be freely done to desired shapes at any head pose without background distortion.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \multicolumn{2}{c}{**Without Correspondence Filtering**} & \multicolumn{2}{c}{**With Correspondence Filtering**} \\ \cline{2-5}  & **Without OpenMP** & **With OpenMP** & **Without OpenMP** & **With OpenMP** \\ \hline Frontal face & 35.70 & 25.77 & 30.15 & 20.63 \\ Oblique face & 97.15 & 60.31 & 68.03 & 30.58 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Processing time (ms) for face model fitting without and with acceleration.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Module** & **Process** & **Time** & **Subtotal Time** \\ \hline \multirow{2}{*}{Registration} & Facial landmark detection & 36.28 \\ \cline{2-3}  & Face model fitting & 22.21 \\ \hline \multirow{2}{*}{Manipulation} & Mesh deformation & 7.14 \\ \cline{2-3}  & Merging fitted and deformed models & 5.34 \\ \hline \multirow{4}{*}{Rendering} & Inpainting & 31.16 \\ \cline{2-3}  & Update textures & 3.37 \\ \cline{2-3}  & Update texture coordinates & 6.49 \\ \cline{2-3}  & Rendering image & 0.36 \\ \cline{2-3}  & Rendering face model & 0.33 \\ \hline Total & & 112.68 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Total processing time (ms).