# ViT-Based Classification and Self-Supervised 3D Human Mesh Generation from NIR Single-Pixel Imaging
## 5 Proposed Method
### Selection of Key Hyperparameters
This might involve a sophisticated deep learning architecture that predicts 3D geometry from 2D data. For training, the CAPE dataset [69] was employed alongside the Adam optimization algorithm, with a learning rate set to \(10^{-4}\). * **Parameter extraction:** Extract key parameters such as global rotation \(R\), translation \(t\), body shape \(\beta\), and pose \(\theta\) from the thresholded depth image, using these to construct and refine the 3D mesh. * **Loss function:** The overall loss function \(L=L_{recon}+L_{pose}+L_{shape}+L_{reg}\) combines multiple components to ensure the accurate reconstruction of the 3D human mesh: (a) **Reconstruction loss (\(L_{recon}\)):** Measures the error between the predicted mesh and the ground truth mesh. This loss encourages the network to produce an accurate 3D reconstruction of the human body. \[L_{recon}=\left\|M_{pred}(R,t,\beta,\theta)-M_{gt}\right\|^{2}\] (7) where \(M_{pred}(R,t,\beta,\theta)\) is the predicted 3D mesh generated using the SM-PLX parameters and \(M_{gt}\) is the ground truth 3D mesh. (b) **Pose loss (\(L_{pose}\)):** Ensures that the predicted pose \(\theta\) closely matches the ground truth pose. This is crucial because the pose directly influences the human body's appearance in 3D space. \[L_{pose}=\left\|\theta_{pred}-\theta_{gt}\right\|^{2}\] (8) where \(\theta_{pred}\) is the predicted pose parameters and \(\theta_{gt}\) is the ground truth pose. (c) **Shape loss (\(L_{shape}\)):** Encourages the predicted body shape \(\beta\) to match the actual body shape from the dataset. This loss helps refine the human mesh to match the body proportions of the subject. \[L_{shape}=\left\|\beta_{pred}-\beta_{gt}\right\|^{2}\] (9) where \(\beta_{pred}\) is the predicted shape parameters and \(\beta_{gt}\) is the ground truth shape. (d) **Regularization loss (\(L_{reg}\)):** To avoid unrealistic poses and shapes, regularization is added to the predicted parameters. This typically includes terms that penalize large deviations from expected values. \[L_{reg}=\lambda_{pose}\cdot\left\|\theta_{pred}\right\|^{2}+\lambda_{shape} \cdot\left\|\beta_{pred}\right\|^{2}\] (10)where \(\lambda_{pose}=0.1\) and \(\lambda_{shape}=0.01\) are hyperparameters that control the regularization strength. 5. **Probabilistic human model fitting**
* **SMPL-X model:** Use the SMPL-X model, which provides a parametric mesh of the human body that can articulate and deform to match the observed data [70], it can be described as a function \(V=M(\theta,\beta,\psi)\), \(\theta\) pose parameters (rotation angles of joints), \(\beta\) shape parameters (capturing the individual's body shape), and \(\psi\) expression parameters (expressions of the face and hand). * **Optimization technique:** The goal is to adjust the SMPL-X parameters \(\theta,\beta,\psi\) so that the model fits the observed depth data from cloud point estimation We can formulate this as a probabilistic optimization problem, minimizing an error function \(E(\theta,\beta,\psi)=E_{data}(M(\theta,\beta,\psi))+\lambda_{reg}\cdot E_{ reg}(\theta,\beta,\psi)\), where \(E_{data}=E_{data}=\sum_{i=1}^{N}\lVert v_{i}-d_{i}\rVert^{2}\) This term measures the error between the predicted model points \(v_{i}\) and the estimated depth points observed \(d_{i}\) from the NIR-SPI image. The term \(E_{reg}=\lVert\beta\rVert^{2}+\lVert\psi\rVert^{2}+\lVert\theta\rVert^{2}\) is used to avoid overfitting the control of deviations from the shape and expression parameters. * **Gaussian correction error pose model estimate:** Pose parameters \(\theta\) can be adjusted iteratively by applying a Gaussian correction based on the error between the observed and predicted joint angles [71]. The updated pose parameters are refined by: \(\theta^{(k+1)}=\theta^{(k)}-\alpha\cdot\bigtriangledown_{\theta}\cdot E\left( \theta^{k},\beta^{k},\psi^{k}\right)\), where \(\alpha\) is the step size, and the error gradient \(E\) is calculated to \(\theta\), allowing for a gradual correction of the model pose.