# ViT-Based Classification and Self-Supervised 3D Human Mesh Generation from NIR Single-Pixel Imaging
## 5 Proposed Method
### Selection of Key Hyperparameters  
The design decisions and hyperparameter configurations throughout the present work were carefully formulated to develop an accurate, efficient, and robust pipeline for reconstructing 3D human poses (see Figure 4). Each component was optimized to achieve an optimal trade-off between computational efficiency and reconstruction fidelity, ensuring seamless integration across the entire pipeline, from data acquisition to model generation. The proposed method achieves enhanced performance by optimizing the hyperparameters, exhibiting strong robustness to noise and high adaptability to low-resolution inputs, while effectively maintaining realistic representations of human poses. The following step-by-step breakdown outlines the entire workflow, encompassing SPI-based image acquisition, depth perception enhancement, and 3D human mesh reconstruction using the SMPL-X model. 1. **Single-Pixel Imaging acquisition (NIR-SPI)** * **Process overview:** Utilize NIR-SPI system to capture an image. SPI systems collect data by measuring the total amount of light reflected from a scene through a series of programmable masks without forming a traditional image (see Algorithm 1). 2. **Depth map** * **Deep Learning model:** Implement Vision Transformer (ViT) to infer a depth map from the SPI image. The ViT would be trained on a dataset [66] and the LWIRPOSE dataset [60] comprising SPI images and their corresponding depth maps, enabling it to predict depth from unseen SPI data. 3. **Output Specifications:** The resulting depth map assigns a depth value to each pixel, encoded as color intensities, illustrating the distance of objects from the imaging setup (see Figure 5). 4. **Depth thresholding** * **Segmentation process:** Apply a thresholding algorithm to the enhanced depth map to isolate the human figure from the background. This binary segmentation helps simplify the input for the subsequent 3D modeling steps [67]. \[S(x,y)= \begin{cases}1,&\text{if }D(x,y)\leq T\\ 0,&\text{if }D(x,y)>T\end{cases} \] (5) where \(D(x,y)\) is the depth value at pixel location \((x,y)\), \(T\) is the threshold value, and \(S(x,y)\) is the resulting binary mask. Figure 4: The image illustrates a comprehensive method for constructing a 3D human model using a NIR-SPI system. The process begins with acquiring an NIR-SPI image and generating a depth map using deep learning techniques. Enhanced depth perception is achieved through additional image processing, which then undergoes thresholding to isolate the human figure. A self-supervised neural network utilizes the processed data to generate a 3D human mesh, extracting crucial parameters such as global rotation, translation, shape, and pose. The final output is a probabilistic human model in the SMPL-X format, offering a precise and articulated 3D representation of the subject captured under challenging imaging conditions. Figure 5: Generation of depth maps using NIR-SPI Images and the LWIRPOSE dataset (**a**) Images from the LWIRPOSE dataset depicting various human poses, depth maps generated from the dataset images and 3D depth perception, (**b**) NIR-SPI images of human subjects, depth maps generated from NIR-SPI images and 3D depth perception at distance 0.5 m, (**c**) NIR-SPI images, depth maps and 3D depth perception at distance 1 m.  
* **Segmentation output:** The output is a point cloud \(P=\{(x_{i},y_{i},z_{i})\}\) where the human figure is distinctly separated. This can be derived from the segmentation mask as: \[P=\{(x_{i},y_{i},z_{i})\mid S(x_{i},y_{i})=1\}\] (6) where \(z_{i}=D(x_{i},y_{i})\) represents the depth at the coordinates \((x_{i},y_{i})\). The output point cloud \(P\) focuses on the points corresponding to the human figure, facilitating more targeted processing in the subsequent 3D modeling steps. 4. **Self-supervised 3D human mesh generation** * **Model architecture:** Use a self-supervised learning framework based on PointNet++ [68] capable of reconstructing 3D human meshes from processed images.