# Robust and High-Fidelity 3D Face Reconstruction Using Multiple RGB-D Cameras
## 3 Materials and Methods
### Multi-View Texture Alignment  
After we solve the problem of geometric accuracy in face reconstruction, we cannot ignore the problem of texture fusion if we want to ensure 3D face model reconstruction with high-quality geometric and visual performance. Although an overlap of textures can be made almost unnoticeable by the pixel weighting, it is not enough to solve the problem of color non-conformity in the captured RGB images caused by different views and individual camera differences.  
Inspired by professional color calibration methods [24], we assume that colors observed from the same spatial point at different views are linear, as shown in Equation (7):  
\[\left[\begin{array}{c}R^{\prime}\\ G^{\prime}\\ B^{\prime}\end{array}\right]=ColorMatrix*\left[\begin{array}{c}R\\ G\\ B\\ 1\end{array}\right] \tag{7}\]  
where the input and output of the expression are the color representation of different pixels corresponding to the same spatial point in images acquired from different views, and \(ColorMatrix\in R^{3\times 4}\) is the color mapping matrix.  
We design the solution as follows. First, we take a captured RGB image of a human face with standard color as a template; then, we use the color transfer [25] in traditional image processing algorithms to align all the image colors to the template to prevent excessive differences between the colors of images captured by different cameras. Then, because we have the aligned RGB-D image pairs, each point in the point cloud has not only its spatial location information but also its corresponding color information. Next, we use the results of the ICP algorithm to obtain the matching point pairs in different views of the point cloud using the following point-matching constraints:  
\[\left\{(p_{i},q_{j^{*}})\middle\|\left\|Rp_{i}+t-q_{j^{*}}\right\|^{2}\leq r^{2 }\right\} \tag{8}\]  
\[j^{*}=\underset{1\leq j\leq N}{\operatorname{argmin}}\big{\|}Rp_{i}+t-q_{j} \big{\|}^{2} \tag{9}\]  
where \(R,t\) are the results obtained by the ICP algorithm; \(p_{i}\) is the \(i\)-th point in the source point cloud (the source point cloud is the point cloud in the left and right views); \(q_{j}\) is the \(j\)-th point in the target point cloud (the target point cloud is the point cloud in the middle view); N is the number of points in the target point cloud; \(j^{*}\) is the label of the nearest point in the target point cloud to \(p_{i}\) in the source point cloud; and r is the hyperparameter used as the upper limit for the distance between \(p_{i}\) and \(q_{j^{*}}\), above which the points cannot be regarded as matching points. Based on the matched point pairs obtained by Equation (8), the corresponding color matching pairs \((p_{i}^{rgb},q_{j^{*}}^{rgb})\) are obtained. \(p_{i}^{rgb}\) represents the color information (RGB value) of the \(i\)-th point in the source point cloud, and \(q_{j^{*}}^{rgb}\) represents the color information of the \(j^{*}\)-th point in the target point cloud. Finally, the corresponding color matching pairs are used to solve the following optimization problem to determine the optimal color matrix \(ColorMatrix^{*}\):  
\[ColorMatrix^{*}=\underset{ColorMatrix}{\operatorname{argmin}}\sum_{j,j^{*}} \big{\|}ColorMatrix*p_{i}^{rgb}-q_{j^{*}}^{rgb}-q_{j^{*}}^{rgb}\big{\|}^{2} \tag{10}\]  
This is a classical least-squares problem [26], which can be efficiently solved with the orthogonal Procrustes algorithm [27; 28] using singular value decomposition (SVD). \(ColorMatrix^{*}\) is then used to adjust the source photo to a new color. This ensures that the left and right view textures are the same color as the middle view textures.