# High-Speed Dynamic Projection Mapping onto Human Arm with Realistic Skin Deformation +
## 3 Proposed Projection Mapping System
### High-Speed Rendering  
The arm surface, which is represented by 3D mesh \(\mathbf{T}\in\mathbb{R}^{3N}\) with \(N\) vertices, is reconstructed using the SMPL model [38]. Although the SMPL model provides high-quality reconstruction, it suffers from various problems. For instance, despite the accurate shape representation [41], small details of skin deformation are missing in the SMPL model due to its linearity [49].  
The first row of Figure 2 illustrates the accuracy problems of the SMPL model. The misalignment between the projected patterns and black crosses is clear. This is a critical problem in projection mapping because users are sensitive to misalignments between the deformed skin and projected images. This problem is more noticeable because the users observe the projected image directly on their arms instead of on a separate display.  
Regarding the arm, wrist pronation, and supination tend to cause the most obvious skin deformations that cause the most severe misalignments, as shown in the first row of Figure 2. Pronation and supination are rotations around the axis of the forearm. Thus, we assume that the misalignments caused by these motions are mainly perpendicular to the axis of the forearm. To improve the accuracy of describing skin deformation in the SMPL model for dynamic projection mapping, we propose an efficient regression-based  
Figure 2: Photographs before (first row) and after (second row) accuracy compensation. The black patterns are projected images. The black crosses drawn on the arm using an ink marker indicate the ground truth for skin deformation. Numbers of crosses between the projected black patterns in the time sequence before and after accuracy compensation (bottom).  
compensation method. The method modifies the texture coordinates of the model in real time as follows:  
\[\begin{cases}\mathbf{u}^{\prime}=\mathbf{g}(\mathbf{v}^{\star},\mathbf{\gamma}_{\mathit{wrist}})+ \mathbf{u}^{\star}\in\mathbb{R}^{N},\\ \mathbf{v}^{\prime}=\mathbf{v}^{\star}\in\mathbb{R}^{N},\end{cases} \tag{1}\]  
where \((\mathbf{u}^{\star},\mathbf{v}^{\star})\) are the texture coordinates provided by the arm-surface model initially before compensation. Respectively, \(u\)-axis is perpendicular to the axis of the forearm and \(v\)-axis is parallel to the axis of the forearm. Regression model \(\mathbf{g}(\mathbf{v}^{\star},\mathbf{\gamma}_{\mathit{wrist}})\) is prepared in advance using polynomial curve fitting of user-specific data. The compensation method provides \(\mathbf{u}^{\prime}\) and \(\mathbf{v}^{\prime}\), the texture coordinates of the model that are consistent with the reconstruction of the arm surface, \(\mathbf{T}\), obtained by the SMPL model.  
In the proposed system, the SMPL model serves as shape reconstruction method, and the compensation method can run in parallel to accelerate processing, as shown in Figure 3. Then, the reconstruction and compensation methods are combined by texture mapping, which maps an image onto a 3D shape. The required processing time of the proposed compensation method is shorter than that of the arm-surface reconstruction due to the simple polynomial regression. Thus, the proposed compensation method does not increase the overall system latency and improves the accuracy of dynamic projection mapping.