SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $H \in \mathbb{R}^{w \times h \times C}$  represent the ground truth hyperspectral cube, and  $\tilde{H} \in \mathbb{R}^{w \times h \times C}$  denote the reconstructed cube, where  $C = 31$  for the 31-band data. In order to focus the evaluation specifically on the facial skin-spectra components within the hyperspectral cube  $H$ , we can utilize a mask  $M \in [0,1]^{w \times h}$  to exclude the background during the assessment. The mask  $M(i,j) = 1$  indicates that the pixel at location  $(i,j)$  corresponds to the human subject, while  $M(i,j) = 0$  signifies the background pixels that should be discarded in the evaluation process. Let  $H_{s} \in \mathbb{R}^{w \times h \times C}$  be the resulting matrix that has all the background removed. To obtain  $H_{s}$ , we need to compute the element-wise multiplication between the mask  $M$  and  $H$  at  $C = k$ , i.e.,
$$
H _ {s} (:,: k) = H (:,: k) \odot M \tag {2}
$$
This element-wise multiplication is repeated for all channels of  $H$  to obtain the final channel-wise multiplication matrix  $H_{s}$ .
Spatial Evaluation The evaluation from the spatial domain focuses on the quality of the reconstructed cube at each band in terms of spatial similarity. For this, we use SSIM to compute the spatial similarity between the ground truth and reconstructed HSI. Let  $h_s^{(P)}$  and  $\tilde{h}_s^{(P)}$  be the patch of the ground truth and reconstructed HSI, then SSIM between each patch can be described as follows:
$$
S S I M \left(h _ {s} ^ {(P)}, \tilde {h} _ {s} ^ {(P)}\right) = l \left(h _ {s} ^ {(P)}, \tilde {h} _ {s} ^ {(P)}\right) ^ {\alpha} c \left(h _ {s} ^ {(P)}, \tilde {h} _ {s} ^ {(P)}\right) ^ {\beta} s \left(h _ {s} ^ {(P)}, \tilde {h} _ {s} ^ {(P)}\right) ^ {\gamma} \tag {3}
$$
where  $\alpha, \beta$  and  $\gamma$  are the weighting parameters for the luminance, contrast, and structural comparison functions. For the detailed formulation of each component, please refer to [40]. To account for mask-edge effects, if a patch is only partially covered by the mask, SSIM calculations consider only the masked pixels. This approach focuses evaluation on relevant information and minimizes the influence of background pixels on SSIM scores. However, in cases where a patch is entirely background, division by zero occurs in SSIM calculation. To avoid mathematical errors, such patches are excluded from evaluation. This ensures meaningful and reliable SSIM scores for informative patches within the mask.
Spectral Evaluation On the other hand, the spectral domain evaluation aims to assess the accuracy of the spectral signature in the reconstructed cube at specific pixel positions. For this, we used SAM to compute the cosine angle between each spectra vector of the ground truth and reconstructed HSI. Let  $\mathbf{h}_{ij} = H(i,j,:)\in \mathbb{R}^C$  be a  $C$ -dimensional spectra vector for the ground truth hyperspectral at location  $i$  and  $j$ , and let  $\tilde{\mathbf{h}}_{ij}$  be the corresponding spectra vector for the reconstructed cube, we can compute the SAM between these 2 spectra vectors as follows:
$$
S A M \left(\mathbf {h} _ {i j}, \tilde {\mathbf {h}} _ {i j}\right) = \cos^ {- 1} \left(\frac {\sum_ {k = 1} ^ {C} \mathbf {h} _ {i j} (k) \tilde {\mathbf {h}} _ {i j} (k)}{\sqrt {\sum_ {k = 1} ^ {C} \mathbf {h} _ {i j} (k) ^ {2}} \sqrt {\sum_ {k = 1} ^ {C} \tilde {\mathbf {h}} _ {i j} (k) ^ {2}}}\right), \tag {4}
$$
where  $\mathbf{h}_{ij}(k)$  denote the pixel value at band  $k$  for a spectra vector located at  $i$  and  $j$  of the HSI cube. Contrary to the perception that the cosine angle restricts similarity, it's important to clarify that SAM computes the inverse cosine, yielding a range of values between 0 and 3.142. This metric is chosen to quantify the spectral alignment between two spectra in an  $n$ -dimensional space, where the dimensionality corresponds to the number of spectral bands [41]. A lower SAM value signifies a better alignment with the reference (ground truth) spectrum.
4.2 Implementation Details and Experimental Results
We conducted the experiment using two datasets prepared in Section 3.2 and evaluated the performance of baseline models (MST++, HRNet, and HSCNN) [42], that were trained with the NTIRE2022 dataset licensed under the GNU General Public License v3.0 [43]. We then proceeded to re-train
Table 3: Spatial Evaluation with SSIM
Data	Pre-trained Models	Re-trained Models
(RGB, VIS)	(MSI, NIR)	(RGB, VIS)	(MSI, NIR)
with	HSCNN [37]	0.683 ± 0.027	-	0.916 ± 0.013	0.943 ± 0.007
Back-ground	HRNet [38]	0.704 ± 0.023	-	0.933 ± 0.021	0.955 ± 0.006
MST++ [39]	0.602 ± 0.042	-	0.923 ± 0.011	0.959 ± 0.006
w/o	HSCNN [37]	0.816 ± 0.021	-	0.950 ± 0.011	0.964 ± 0.006
Back-ground	HRNet [38]	0.813 ± 0.023	-	0.961 ± 0.014	0.971 ± 0.005
MST++ [39]	0.766 ± 0.035	-	0.954 ± 0.010	0.974 ± 0.004
Figure 3: The top-right diagram represents the camera response function used to produce the RGB image. We conducted spectral evaluation across the 31 bands of images. The evaluation of the pre-trained HSCNN model's spectral performance reveals relatively better reconstruction in the middle bands (460nm - 520nm) as compared to the bands at the two ends. Notably, the spatial evaluation also shows that the skin areas in those middle bands exhibit better SSIM compared to the last few bands.
460nm, SSIM: 0.892
520nm.SSIM:0.824
580nm,SSIM:0.810
470nm, SSIM: 0.904
530nm,SSIM:0.821
480nm, SSIM: 0.915
540nm, SSIM: 0.799
490nm, SSIM: 0.913
550nm, SSIM: 0.779
these models with our Hyper-Skin dataset for 100 epochs using an RTX5000 GPU and 16-bit memory. To address memory constraints, we randomly cropped the input size to  $128 \times 128$  during training, while the entire  $1024 \times 1024$  image was used for testing. The hyperparameters used for re-training the models remained the same, except for reducing the number of HSCNN blocks to 20 for the (MSI, NIR) experiments and adjusting the number of input channels to 4 for the (MSI, NIR) pair of data. Adam optimizer with a learning rate of 0.0004 was employed for training all three models.
4.2.1 Spatial Domain
Table 3 presents the spatial evaluation results using SSIM for two types of data: (RGB, VIS) and (MSI, NIR). The evaluation was conducted with and without the background, where the background was removed using a mask to focus on the human subject. Figure 3 provides a result illustration with pre-trained HSCNN. The pre-trained models were only applied to the (RGB, VIS) data pair since they were trained on the NTIRE dataset, which focuses on hyperspectral reconstruction in the visible spectrum. The (MSI, NIR) data pair requires an additional channel, which is not supported by the pre-trained models, thus they were not used for evaluation. After re-training the models with our Hyper-Skin dataset, significant improvements in performance were observed, as indicated in Table 3. Comparing the results with and without the background, it is evident that most of the reconstruction issues are associated with the background. For applications that focus on human skin, such as hyperspectral skin analysis, the performance of skin reconstruction is crucial. The
Table 4: Spectral Evaluation with SAM
Data	Pre-trained Models	Re-trained Models
(RGB, VIS)	(MSI, NIR)	(RGB, VIS)	(MSI, NIR)
with	HSCNN [37]	0.677 ± 0.061	-	0.119 ± 0.008	0.091 ± 0.010
Back-ground	HRNet [38]	0.648 ± 0.062	-	0.147 ± 0.014	0.094 ± 0.009
MST++ [39]	0.707 ± 0.054	-	0.113 ± 0.009	0.086 ± 0.006
w/o	HSCNN [37]	0.621 ± 0.049	-	0.113 ± 0.009	0.083 ± 0.012
Back-ground	HRNet [38]	0.596 ± 0.046	-	0.133 ± 0.015	0.086 ± 0.010
MST++ [39]	0.628 ± 0.050	-	0.107 ± 0.010	0.076 ± 0.005
\*Note: the pre-trained model based on NTIRE dataset is defined to take in the 3-channel input, and cannot be applied to MSI data that has 4-channel input.
Figure 4: Re-training all three models with our Hyper-Skin dataset yielded a notable enhancement in performance, particularly in the skin area. However, it is worth noting that the first few bands of the reconstructed cube may not fully capture the variations present in the skin.
Figure 5: The demonstration of a real selfie image captured with a smartphone shows that the trained model is capable of reconstructing spectral information at the skin location.
results demonstrate that the reconstruction of the skin area exhibits better performance, supporting the potential for low-cost hyperspectral skin solutions, such as reconstructing the facial skin-spectra cube from smartphone selfies.
4.2.2 Spectral Domain
Table 4 shows SAM-based spectral evaluation results for two data types: (RGB, VIS) and (MSI, NIR), with and without background. SAM ranges from  $\pi /2$  to 0, in which closer to 0 indicating better reconstruction. After re-training models using our Hyper-Skin dataset, both data types showed significant performance improvements, as illustrated in Figure 4. Note that the performance of the models was generally lower when the background was present compared to when it was removed. This highlights the impact of background interference on the reconstruction results. These findings emphasize the potential of our dataset in enhancing hyperspectral reconstruction for applications related to skin analysis and other fields that require accurate spatial information. Note that the re-trained models also showed good performance on the (MSI, NIR) data. We further verify the reconstruction performance on the real RGB image taken by a smartphone. As shown in Figure 5, the reconstruction model is capable of estimating the skin spectral information. Due to page constraints, the visualizations of these results are provided as supplementary materials.
5 Limitations, Ethical Considerations and Societal Impact
We worked closely with the university's ethical review board to ensure the inclusivity, privacy, and ethical integrity of the Hyper-Skin dataset. This ensured responsible data usage for skin analysis research. Our data collection followed strict participant consent procedures, including providing participants with detailed project information before obtaining written and verbal consent. The ethics review protocol and consent procedures are provided as supplementary materials.
**Limitations** Our dataset's representativeness might be limited, potentially not capturing the full diversity of skin types, tones, and conditions across various populations. This could introduce biases and hinder model generalization. Additionally, while our deep neural network-based approach leverages the dataset for latent priors, it's constrained by dataset limitations. Novel cosmetology conditions not covered in training might affect model performance due to distribution shifts, a common challenge in machine learning [44, 45], particularly in medical datasets with skewed distributions [46, 47]. To address this, we've secured ethical approval to expand data collection, targeting diverse participants and cosmetology conditions to enhance practical utility.
Ethical Considerations To ensure ethical compliance throughout the data collection process, we implemented measures to anonymize personally identifiable information, such as assigning a subject ID instead of using participants' real information. Robust security measures were also put in place to protect sensitive data from unauthorized access or misuse. We strictly followed guidelines provided by the research ethics board and obtained informed consent from every participant, respecting their autonomy and ensuring they understood how their data would be used.
Societal Impact Our Hyper-Skin dataset revolutionizes skin analysis by providing affordable and accessible solutions. With its ability to reconstruct skin spectral properties and estimate parameters like melanin and hemoglobin concentration, it empowers researchers and practitioners to develop low-cost skin analysis solutions directly for consumers. The dataset's societal impact extends to individuals monitoring their skin's well-being and skincare companies developing personalized products and innovative AI models. By driving advancements in skin analysis, the Hyper-Skin dataset benefits individuals, professionals, and the skincare industry as a whole.
This paper contributes to the field of hyperspectral skin analysis by providing a comprehensive collection of facial skin hyperspectral data, named Hyper-Skin dataset. The novelty of this dataset lies in its spectral coverage in the VIS and NIR spectrum, offering potential applications in skin monitoring and customized cosmetic products at the consumer's fingertips. It serves as a valuable resource for algorithm development and evaluation, with future directions including dataset diversification, advanced analysis techniques, and interdisciplinary collaborations, inviting researchers and practitioners to contribute to the advancement of hyperspectral skin analysis for human well-being.
[1] F. Vasefi, N. MacKinnon, R. B. Saager, A. J. Durkin, R. Chave, E. H. Lindsley, and D. L. Farkas, "Polarization-sensitive hyperspectral imaging in vivo: a multimode dermoscope for skin analysis," Scientific reports, vol. 4, no. 1, pp. 1-10, 2014.
[2] L. Gevaux, J. Gierschendorf, J. Rengot, M. Cherel, P. Séroul, A. Nkengne, J. Robic, A. Trémeau, and M. Hébert, "Real-time skin chromophore estimation from hyperspectral images using a neural network," Skin Research and Technology, vol. 27, no. 2, pp. 163-177, 2021.
[3] G. N. Stamatas, B. Z. Zmudzka, N. Kollias, and J. Z. Beer, "Non-invasive measurements of skin pigmentation in situ," Pigment Cell Research, vol. 17, no. 6, pp. 618-626, 2004.
[4] R. Koprowski, S. Wilczyński, Z. Wrobel, and B. Błonska-Fajfrowska, “Calibration and segmentation of skin areas in hyperspectral imaging for the needs of dermatology,” Biomedical engineering online, vol. 13, no. 1, pp. 1-19, 2014.