Hyper-Skin: A Hyperspectral Dataset for Reconstructing Facial Skin-Spectra from RGB Images
Authors: Pai Chet Ng
Zhixiang Chi
Yannick Verdie
Juwei Lu
Konstantinos N Plataniotis
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: hyperspectral skin analysis
skin-spectra reconstruction
hyperspectral dataset
TL;DR: Introducing Hyper-Skin, a comprehensive hyperspectral dataset enabling facial skin-spectra reconstruction on consumer devices.

Abstract: We introduce Hyper-Skin, a hyperspectral dataset covering wide range of wavelengths from visible (VIS) spectrum (400nm - 700nm) to near-infrared (NIR) spectrum (700nm - 1000nm), uniquely designed to facilitate research on facial skin-spectra reconstruction.
By reconstructing skin spectra from RGB images, our dataset enables the study of hyperspectral skin analysis, such as melanin and hemoglobin concentrations, directly on the consumer device. 
Overcoming limitations of existing datasets, Hyper-Skin consists of diverse facial skin data collected with a pushbroom hyperspectral camera. 
With 330 hyperspectral cubes from 51 subjects, the dataset covers the facial skin from different angles and facial poses.
Each hyperspectral cube has dimensions of 1024$\times$1024$\times$448, resulting in millions of spectra vectors per image. 
The dataset, carefully curated in adherence to ethical guidelines, includes paired hyperspectral images and synthetic RGB images generated using real camera responses. 
We demonstrate the efficacy of our dataset by showcasing skin spectra reconstruction using state-of-the-art models on 31 bands of hyperspectral data resampled in the VIS  and NIR spectrum. 
This  Hyper-Skin dataset would be a valuable resource to NeurIPS community, encouraging the development of novel algorithms for skin spectral reconstruction while fostering interdisciplinary collaboration in hyperspectral skin analysis related to cosmetology and skin's well-being. 
Instructions to request the data and the related benchmarking codes are publicly available at: https://github.com/hyperspectral-skin/Hyper-Skin-2023.

Hyper-Skin: A Hyperspectral Dataset for Reconstructing Facial Skin-Spectra from RGB Images
Pai Chet Ng , Zhixiang Chi 1 Yannick Verdie 2, Juwei Lu Konstantinos N. Plataniotis
1 The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto
2 Huawei Noah's Ark Laboratory, Huawei Canada
Abstract
We introduce Hyper-Skin, a hyperspectral dataset covering wide range of wave- lengths from visible (VIS) spectrum (400mm 700nm) to near-infrared (NIR) spectrum (700mm 1000nm), uniquely designed to facilitate research on facial skin-spectra reconstruction. By reconstructing skin spectra from RGB images, our dataset enables the study of hyperspectral skin analysis, such as melanin and hemoglobin concentrations, directly the consumer device. Overcom- ing limitations of existing datasets, Hyper-Skin consists of diverse facial skin data collected with pushbroom hyperspectral camera. With 330 hyperspectral cubes from 51 subjects, the dataset covers the facial skin from different angles and facial poses. Each hyperspectral cube has dimensions of 1024x 1024x448,
resulting in millions of spectra vectors per image. The dataset, carefully cu- rated in adherence to ethical guidelines, includes paired hyperspectral images and synthetic RGB images generated using real camera responses. We demon- strate the efficacy of our dataset by showcasing skin spectra reconstruction using state-of-ther of-the-art art models on 31 bands of hyperspectral data resampled in the VIS and NIR spectrum. This Hyper-Skin dataset would be a valuable resource NeurIPS community, encouraging the development of novel algorithms for skin hyper-
spectral reconstruction while fostering interdisciplinary collaboration spectral skin analysis related to cosmetology and skin 's well-being. Instructions request the data and the related benchmarking codes are publicly available atttps///githee
http://yyyerrperral-kkhhtteerrr Skin- 2023
1 Introduction
Hyperspectral imaging offers comprehensive and non-invasive approach for facial skin analysis,
capturing detailed spatio-spectral information across wide range of wavelengths 21 This three-
dimensional hyperspectral cube surpasses the limitations of single-point measurements, providing deeper understanding of facial skin characteristics and spatial distribution [3]. Previous studies
have demonstrated the potential of hyperspectral skin analysis in dermatology 4 cosmetics (5), and skin's 's well-being (61. paving the way for advanced analysis and applications in these domains. This paper introduces "hyper-skin" a hyperspectral skin dataset uniquely designed to facilitate the development of algorithms targeting consumer- bassdd cosmetology applications. This unique dataset is curated with this specific goal in mind, focusing its practical relevance within the consumer- based cosmetology and skin beauty.
Despite the potential of hyperspectral skin analysis on cosmetology and skin beauty, the high cost and limited accessibility of hyperspectral imaging systems have limited their widespread adoption. Consumer cameras, particularly those embedded smartphones, have become an integral part of daily life and are extensively used for capturing selfies and everyday images. Hence, many works While RGB
study the use of RGB images from consumer cameras for skin analysis 7
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Vi5 specirrum of skin at pizel ocation (500, 500) (510, 51.0
NIR spectrurm of skin pizel location (500, 50C) (510. 510)
0.75
refectunc
85-
odhectnce
0.65
0.60
0.35
82-
(520
0.50
600
650
S00
#50
wereelenth trmn
wavelength Enmo
VIS
100000
NIR
Figure 1: A glimpse of our Hyper-Skin dataset, covering the skin spectra in the visible spectrum (400nm 700mm) and near-infrared spectrum (700mm 1000mm).
images have been used for certain skin analysis tasks 10 [1] they lack the ability capture the comprehensive spatio-spectral information provided by hyperspectral imaging, limiting the depth of skin analysis. light of the prevalence of consumer cameras, an intriguing idea emerges: Can we reconstruct valuable information from expensive hyperspectral cubes using accessible RGB images, enabling hyperspectral skin analysis directly on consumer devices?
This highlights the need for a comprehensive dataset to develop computational reconstruction methods for the question above. While RGB datasets such those from the International Skin Imaging Collaboration (ISIC) competition series (2016 2020) capture visual information, they lack the corresponding hyperspectral data required for studying hyperspectral reconstruction [1] 13 [4]. On the other hand, hyperspectral datasets enable the exploration of relationships between skin spectra and spatial distribution. Although the RGB counterpart can synthetically generated from a given hyperspectral cube using known camera response function, publicly available hyperspectral datasets focusing specifically on facial skin analysis are limited and often inaccessible to the public. Furthermore, existing hyperspectral datasets primarily focus on the visible (VIS) spectrum (400nm 700nm), disregarding the valuable near-infrared (NIR) spectrum (700mm 1000nm). These limitations highlight the necessity for hyperspectral dataset that addresses these gaps and facilitates the development of low-cost and accessible hyperspectral skin analysis on consumer devices.
Our Contributions Our Hyper-Skin dataset uniquely designed to unlock the potential of hy- perspectral skin analysis directly on the consumer device. With high spatial and spectral resolution, ie., x024x 1024 x448, Hyper-Skin offers an extensive collection of hyperspectral cubes, yielding million spectra per image. Notably, offer synthetic RGB images synthesized from 28 real camera response functions, allowing for versatile experimental setups. What sets Hyper-Skin apart is its comprehensive spectral coverage, including both the VIS and NIR spectrum, facilitating holistic understanding of various aspects of human facial skin, enabling new possibilities for consumer applications to see beyond the visual appearance of their selfies and gain valuable insights into their skin's physiological characteristics, such as melanin and hemoglobin concentrations.
2 Related Work
The potential hyperspectral solutions in the skin-related analysis have encouraged the curation of hyperspectral datasets. This section reviews existing hyperspectral datasets related to skin analysis, as summarized in Table and reconstruction aiming to provide affordable hyperspectral solutions accessible to consumers.

---

Table 1: Comparison between Skin-related Data
Work
Exporimental Subjects
Number of Subjects
Spectral Range
Spectral Resolution
Spatial Resolution
Acquisition
Device
Ours
Full face
51
400- 1000 nm
1.34 nm
1024x x 1024
Specim PX10
Arm
857
NA
SkinSpect dermoscoppe
Facial skin
204
400 700
3.3 nm
1148 948
SpectraCamD
1k
Skin losions
330
465 630 nm
10.3125 nm
x 272
Ximca MQU22HG-IM-SN IMSSMAA-VIS
16
Hairless mice
10
500 660 um
NA
NA
Oxy -2, HyperMed
P
Spine
17
450 950
12.2
500 250
snagshot hyperspectral imaging (HSI) camern
18
Fore arm
80
380 1055 nm
2.8 nm
450 x 1310
Specim80 Spectral Camera PS V1OE
Hand
45
397 1030 am
0.79 mm
899 x 1312
PED-VIOE line-scun camera
Skin reflectunce
144
250 nm
Mouse
100
500 1000 am
5
640 480
custou-buiit LCTF
Dermoscopy images, ** Reflectance images, *** Synthetic reflectance
Skin-related Datasets SpectraCam and SpectraFace hyperspectral cameras 20 has been used to collect the data of normal and pathological skin with a spectral resolution of 31 wavelengths in
the VIS spectrum [1] The hyperspectral dermoscopy dataset consists of 330 images, including 80 melanoma images. 180 dysplastic nevus images, and 70 images of other skin lesions, with a
spatial resolution of 512x272 pixels and 16 spectral bands ranging from 465nm to 630nm (15) Hyperspectral dataset of 20 nude mice was collected by [16] as an alternative to human skin [22] to study acute changes in oxygenation and perfusion in irradiated skin. For markerless tracking in spinal navigation, 17 captured hyperspectral images of the skin from 17 healthy volunteers, with spatial resolution of x080 2048 and 41 spectral bands in the VISt NIR range (450-950nm).
Both work in [18] and 4 used the same Specim Spectral Camera PS V10E to acquire hyperspectral data covering the visible to NIR range (380- (500-1055m)) with 1040 bands and a spatial resolution of 450x 1310. The former dataset by 18, containing data from 80 subjects, focused on vein localization, whereas used the dataset for routine dermatological examinations. While references involve capturing NIR spectral information, it's important to note that these datasets are not publicly accessible. Despite the potential of hyperspectral imaging in skin-related applications, most of these existing datasets relied on expensive imaging systems with primary focus on scientific applications, and no efforts were made to provide low-cost hyperspectral solutions for consumer devices.
[18 17
Skin Spectral Reconstruction Datasets Although datasets specitically focused on skin spectra reconstruction are limited, there have been notable contributions in this field. One such dataset is the Skin Hyperspectral Reflectance Database (SHRD) 19, which includes 144 skin directional- hemispherical curves obtained through novel hyperspectral light transport model. This dataset provides valuable insights into reconstructing hyperspectral information on human skin. Another study explores the reconstruction of hyperspectral information in mice skin (9), using 26 SKH-1 hairless albino mice as a model system (2 The researchers propose a mathematical approach to reconstruct hyperspectral data from RGB images, allowing visualization of hemoglobin content across a large skin area without the need for expensive hyperspectral imaging systems. However, these skin-spectral reconstruction datasets have limitations in terms of sample size and spatial resolution, particularly in their coverage of facial images from various angles and poses. Consequently, progress in skin-spectral reconstruction has been relatively slower compared to hyperspectral reconstruction on natural scenes or everyday objects.
Natural Scenes and Everyday Objects Reconstruction Datasets Compared proprictary and unavailable skin-related datasets, several hyperspectral datasets on natural scenes and everyday objects have been made publicly available for the study of hyperspectral reconstruction. The CAVE dataset consists of 32 scenes with spatial resolution of 512 x 512 pixels. 31 spectral bands ranging from 400nm to 700nm at 10nm intervals
(24). The HARVARD dataset includes 50 images captured under daylight illumination, with 31 spectral bands spanning from 420nm to 720nm, and spatial resolution of 464x346 pixels 255. The KAIST dataset comprises 30 hyperspectral images with
spatial resolution of 3376 x 2704 pixels and spectral range from 420nm to 720mm (26). The
3

---

KAIST-depth dataset features 16 indoor scenes with a spectral range of 420nm to 680nm, 27 spectral channels, and spatial resolution of 2824 x 4240 pixels [27]. The New Trends in Image Restoration and Enhancement (NTIRE) series of datasets, including NTIRE2018, NTIRE2020, and NTIRE2022, have significantly advanced spectral reconstruction from RGB images, with varying sizes, spectral resolutions, and spectral ranges [28] [9] BT]. Despite primarily focusing on natural scenes and generic objects, these publicly available datasets offer valuable resources, including facilitating the
development of pre-trained models, that can be extended to skin spectral reconstruction tasks.
3 Hyper-Skin Data Curation and Preparation
This section outlines the methodology we employed for collecting facial skin data and provides a detailed description of our carefully curated Hyper-Skin dataset.
subject
40 cm
FX 10
Halogen light
Ha Halagen light
Figure 2: The image on the right illustrates our experimental setup for data collection, while the accompanying schematic representation provides visual depiction of the setup.
3.1 Data Collection
The data collection process was conducted carefully, taking into account the setup of devices and recruitment of participants while ensuring adherence to the university's 's research ethics protocol. We successfully recruited participants who contributed a total of 306 hyperspectral data. To maintain the privacy and sensitivity of the human subjects involved, we have implemented credentialization procedure to access the dataset. Interested users will be required to digitally sign an End User License Agreement (EULA) online, which outlines the terms and conditions for using the dataset, including provisions for using only the authorized image in future publications. Detailed instructions for requesting the dataset will be publicly available in our GitHub repository, where users can find digital EULA form to facilitate the data access request. Once the EULA form is signed and submitted, users will receive secure link via email to download the data within 24 hours.
Data Acquisition Devices and Set Up The Hyper-Skin dataset was obtained using a Specim FX10 camera, covering 448 spectral bands from 400nm to 1000nm. Consider multiple factors, including participant safety, image quality, and spectral resolution, we opted to use a pushbroom camera rather than the Liquid Crystal Tunable Filter (LCTF) system used by [32]. The camera was moved using a customized scanner for precise scanning, as shown in Figure 1. The distance between the camera and the face was set at 40em, providing spatial resolution of 1024x 1024 pixels. The scanner and camera were controlled by a computer running LUMO recorder software. Further setup details are available in the supplementary material. With a frame rate of 45Hz for one line, it took approximately 22.7 seconds to capture all 1024 lines. To minimize artifacts from line scanning, participants used a chin rest for stability. Halogen lamps illuminated the scene across the visible to near-infrane saar-infrrred spectrum. Since ensuring participant safety (particularly for the eyes area) was top priority. particularly for their eyes, the illumination level of the halogen lamps is carefully adjusted following the manufacturer's advice to prevent any risk to participants' eyes. Using the Specim FX10 camera resulted high-quality images, setting our dataset apart from the mentioned dataset [32] which contains noisy and blurry images that affect skin texture visibility. Differing from the CMU dataset (32), which features a 10nm step size for 65 bands, our dataset encompasses both VIS and NIR spectrum with finer resolution.

---

Table 2: Hyper-Skin data pairs
Description
(RGB, VIS)
(MSI, NIR)
Input
RGB
MSI (RGB + Infrared at 960nm)
Output
VIS (400nm 700nm)
NIR (700mm 1000nm)
Skin physiological features
surface-level characteristies (e.g., pig- mentation and melanin map)
deeper tissue properties (c.g., collagen content and hemoglobin map)
Data Acquisition Process Participants were recruited through online forums and email advertise- ments, and their participation involved signing an informed consent form in accordance with the human research ethics protocol. The approved ethics protocol can be found in the supplementary materials. During the data acquisition process, participants were seated on a stool and asked to rest their face on a chin rest while maintaining stillness. Initially, participants were instructed to have a neutral facial expression, and three face images were captured from different viewpoints (front, left, and right) by rotating the chin rest. This process was then repeated with participants instructed to smile. A total of six images were collected for each participant. Throughout the camera scanning. halogen light remained on. It's worth noting that even with minimal participant movement, slight shifting may occur as the FX10 camera scans line by line. To ensure high-quality images, the captured images were manually inspected by the investigator, and if any shifting was observed, the image was retaken until satisfactory results were achieved. Throughout the entire process, participant anonymity and confidentiality were strictly maintained.
Participants Demographic and Cosmetology Condition Our data collection campaign attracted 51 participants, most participants are in their early 20s and 30s, with a smaller representation from other age groups (10s, 40s-50s). Male participants slightly outnumbered females, potentially due to the gender distribution in the Department of Electrical and Computer Engineering. The majority of participants identified as Asian, with smaller number identifying European or Latino. To improve the generalizability of our findings, we have applied for an extension of our research ethics protocol conduct another data collection next year, aiming to include more diverse sample.
3.2
Data Preparation
The Hyper-Skine dataset was created by collecting RAW hyperspectral data, which were then radio- metrically calibrated and resampled into two separate 31 31-band datasets. One dataset covers the visible spectrum from 400nm to 700nm, while the other dataset covers the near-infrared spectrum from 700nm to 1000mm. Additionally, synthetic RGB and Multispectral (MSI) data were generated, includ- ing RGB images and an infrared image at 960nm. The Hyper-Skin dataset consists of two types of data: (RGB, VIS) and (MSI, NIR), offering different skin analysis capabilities. The visible spectrum data allows for the analysis of surface-level skin characteristics, such as melanin concentration, blood oxygenation, pigmentation, and vascularization. On the other hand, the near-infrared spectrum data enables the study of deeper tissue properties, including water content, collagen content, subcutaneous blood vessels, and tissue oxygenation. As summarized in Table 2 by providing these two distinct ranges of hyperspectral data, the Hyper-Skin dataset caters to different needs in skin analysis and facilitates comprehensive investigations of various skin features.
Data Preprocessing We applied radiometric calibration on the RAW hyperspectral data to extract spectral reflectance information. This involved capturing a white reference image, representing spectrally neutral surface with consistent reflectance values across all bands. A dark reference image was also obtained by closing the camera lens during capture. For precise calibration, selecting an appropriate white reference was crucial. After consultation with the camera vendor, we opted for cost-effective Teflon instead of Spectralon panels, as provided satisfactory spectral response. The preprocessing steps included subtracting dark reference values to eliminate noise, and dividing by white reference values to normalize and convert data to reflectance values, yielding the desired spectral reflectance data.
RGB and MSI data Generation The raw hyperspectral cube with 448 bands was resampled into two sets of 3 ba-d data using SciPy's interpolation function. This downsampling to 31 bands
5

---

is in line with existing practices in hyperspectral reconstruction studies, similar to the CAVE and NTIRE2018-285 8-2022 datasets. strikes a balance between data richness and size. This approach retains hyperspectral differentiation and computational efficiency for analysis, making the data more accessible compared to the 448-band dataset, which exceeds 1TB size. While the complete 448-band dataset is substantial in terms of size, we are prepared to provide it upon specific request. The availability of the 31-band data addresses data transfer constraints, while the backup 448-band data ensures comprehensive access to the dataset.
For realistic RGB data generation, we adopted the HSI2RGB simulation pipeline based on ideal color-matching functions as outlined in [33]. Our emulation of consumer camera-captures images incorporates 28 camera response functions from [34] and [35], encompassing various cameras like DSLR and smartphones. Further details on the measurement setup and gathering camera spectral sensitivity information can found in [4.. While we do possess actual RGB data captured by smartphone sensors, their current utility is constrained by concerns related to alignment quality issues stem from variations in camera models, viewing angles, and overlapping fields of view. In pursuit of maintaining high-quality aligned pairs, we have chosen to provide synthetic RGB images that are perfectly aligned with their hyperspectral counterparts. This approach aligns with existing practices such those in the 2ITRE2018-2075 challenges.
Data Description We intentionally chose 4 participants from the total of 51 to form the testing data. Each participant contributed 6 images, covering 2 facial expressions and 3 face poses. This selection ensured comprehensive representation of facial poses and expressions in the testing dataset. The choice of these 4 participants was deliberate, based on their explicit consent for image use in publications, adhering ethical standards. The remaining data was exclusively used for offline training. Collecting diverse participant images encompassed a variety of natural facial poses and expressions observed in selfies. Opting for a cetticipant-sppcciin approach, rather than random partitioning. prevents data overlap within the same participant in the training set, reducing potential bias. This strategic participant selection safeguards dataset integrity and subsequent analysis.
4 Evaluation and Benchmarks
This section discusses the benchmark design with the facial skin-spectra reconstruction and then presents the experimental results from both the spatial and spectral domains.
4.1 Facial Skin-spectra Reconstruction
The facial skin-spectra reconstruction task focuses reconstructing the hyperspectral cube of the facial skin using the provided RGB image. Given a pair of RGB data represented as R E Ruxxxx and the hyperspectral cube denoted as H E Rwxxxx where << C, the objective of the reconstruction task can be formulated as follows:
= f(;
where the goal is to find the function f(: θ) parameterized by θ that maps the RGB data R to the hyperspectral cube H. Given the extensive research on hyperspectral reconstruction for natural scenes or everyday objects, we can leverage existing hyperspectral reconstruction models as baseline models for our specific facial skin-spectra reconstruction problem.
4.1.1 Baseline Models
Numerous methods have been developed to address hyperspectral reconstruction from RGB images. Interested readers can refer to the survey by (36) for list of representative models that have been developed over the past two decades. For our benchmark design, we specifically consider three models, i.e., Hyperspectral Convolutional Neural Network (HSCNN) (37], Hierarchical Regression Network (HRNet) (38), and Multi-stage spectral-wise transformer (MST++) (39, that have emerged as winners in the NTIRE competition series held in conjunction with CVPR from the year of 2018, 2020 to 2022, respectively.

---

4.1.2 Evaluation Metrics
We consider two types of metries: Structural Similarity Index (SSIM) for spatial evaluation and Spectral Angle Mapper (SAM) for spectral evaluation. Our emphasis lies assessing the facial skin spectra reconstruction in human subjects, excluding the background image from analysis. This approach allows us to precisely gauge physiological properties like melanin and hemoglobin concentrations, essential for effective hyperspectral skin analysis.
Let H E _ represent the ground truth hyperspectral cube, and H E R denote the reconstructed cube, where C' 31 for the 31-band data. In order focus the evaluation specifically on the facial skin-spectra components within the hyperspectral cube H, we can utilize mask indicates that the pixel at location (i.j) corresponds to the human subject, while Mi,j) = 0 signifies the background pixels that should be discarded in the evaluation process. Let H₂ E RxxxxC be the
M E [0, 1] to exclude the background during the assessment. The mask M(..j
resulting matrix that has all the background removed. 'o obtain H,, need compute the element-wise multiplication between the mask M and H at C' = k, i.c.,
H,(;,;,) k) (t,;; k) O M
(2)
This element wise multiplication is repeated for all channels of H to obtain the final channel-wise multiplication matrix H..
Spatial Evaluation
The evaluation from the spatial domain focuses on the quality of the recon- structed cube at each band in terms of spatial similarity. For this, we use SSIM to compute the spatial similarity between the ground truth and reconstructed HSL. Let (P) and P! be be the patch of the ground truth and reconstructed HSI, then SSIM between each patch can be described as follows:
SSTM( ( = = ((NP)), ((P)))((h!)) ((h!)) ,,!!))77
where a, and y are the weighting parameters for the luminance, contrast, and structural comparison functions. For the detailed formulation of each component, please refer to [40]. To account for mask-edge effects, a patch is only partially covered by the mask, SSIM calculations consider only the masked pixels. This approach focuses evaluation on relevant information and minimizes the influence of background pixels on SSIM scores. However, cases where patch entirely background, division by zero occurs in SSIM calculation. To avoid mathematical errors, such patches are excluded from evaluation. This ensures meaningful and reliable SSIM scores for informative patches within the mask.
Spectral Evaluation On the other hand, the spectral domain evaluation aims to assess the accuracy of the spectral signature in the reconstructed cube at specific pixel positions. For this, we used SAM to compute the cosine angle between each spectra vector of the ground truth and reconstructed HSI. Let h. Hi, :)< E Rº be C-dimensional spectra vector for the ground truth hyperspectral at location and j, and let hig be the corresponding spectra vector for the reconstructed cube, we can compute the SAM between these 2 spectra vectors as follows:
De by (k)
SAM (baj, (alyyhh.) b2)) = cos-
V EC (,)( / EE his
where hiy(k) denote the pixel value at band i for spectra vector located at and of the HSI cube. Contrary to the perception that the cosine angle restricts similarity, it's important to clarify that SAM computes the inverse cosine, yielding range of values between 0 and 3. 3.142. This metric is chosen to quantify the spectral alignment between two spectra in an re-iimensional space, where the dimensionality corresponds to the number of spectral bands 4T. A lower SAM value signifies better alignment with the reference (ground truth) spectrum.
4.2 Implementation Details and Experimental Results
We conducted the experiment using two datasets prepared in Section 3.2 and evaluated the perfor- mance of baseline models (MST++, HRNet, and HSCNN) 42), that were trained with the NTIRE2022
datasct licensed under the GNU General Public License v3.0 [43). We then proceeded to re-train
7

---

Table 3: Spatial Evaluation with SSIM
Pre-trained Models
Re Re-trained Models
Data
(RGB, VIS)
(MSI, NIR)
(RGB, VIS)
(MSI, NIR)
with
HSCNN 37
0.683 0.027
0.916 0.013
0.943 = 0.007
Back-
HRNet
0.704 0.023
0.933 0.021
0.955 + 0.006
ground
+ST++
0.602 + 0.042
0.923 + 0.011
0,959 + 0.006
w/o
HSCNN 37
0.816 0.021
0.950 0.011
0.964 1 0.006
Back-
HRNet
38
0.813 1 0.023
0.961 1 0.014
0.971 + 0.005
ground
MST++
39
0.766 + 0.035
0.954 + 0.010
0.974 = 0.004
40hms 22M
400mn 5310
420mn 29M-3813
422ems 5592
440mm, SEM:
450em 5090
Eantur parts
Figure 3: The top-right diagram represents camera response function used to produce the RGB image. conducted spectral evaluation across the 31 bands of images. The evaluation of the pre-trained HSCNN model's spectral performance reveals relatively better reconstruction in the middle bands (460mm 520nm) as compared to the bands at the two ends. Notably, the spatial evaluation also shows that the skin areas in those middle hands exhibit better SSIM compared the last few bands.
these models with our Hyper-Skin dataset for 100 epochs using an RTX5000 GPU and 16-bit memory. To address memory constraints, randomly cropped the input size to 128x 128 during training, while the entire 1024x 1024 image was used for testing. The hyperparameters used for re-training the models remained the same, except for reducing the number of HSCNN blocks 20 for the (MSI, NIR) experiments and adjusting the number of input channels to 4 for the (MSI, NIR) pair data. Adam optimizer with learning rate of 0.0004 employed for training all three models.
4.2.1 Spatial Domain
Table 3
presents the spatial evaluation results using SSIM for two types of data: (RGB, VIS) and (MSI, NIR). The evaluation was conducted with and without the background, where the background was removed using mask to focus on the human subject. Figure 3 provides a result illustration with pre-trained HSCNN. The pre-trained models were only applied to the (RGB, VIS) data pair since they were trained on the NTIRE dataset, which focuses on hyperspectral reconstruction in the visible spectrum. The (MSI, NIR) data pair requires an additional channel, which not supported by the pre-trained models, thus they were not used for evaluation. After re-training the models with our Hyper-Skin dataset, significant improvements
performance were observed, as indicated in Table Comparing the results with and without the background, it is evident that most of the reconstruction issues are associated with the background. For applications that focus on human skin, such as hyperspectral skin analysis, the performance of skin reconstruction is crucial. The

---

Table 4: Spectral Evaluation with SAM
Pre-trained Models
Re- -e--trined Models
Data
(RGB, VIS)
(MSI, NIR)
(RGB, VIS)
(MSI, NIR)
with
HSCNN 37
0,677 0.061
0.119 0.008
0.091 = 0.010
Back-
HRNet
0.648 0.062
0.147 0.014
0.094 + 0.009
ground
+ST++
0.707 + 0.054
0.113 + 0.009
0.086 + 0.006
w/o
HSCNN 37
0,621 0.049
0.113 0.009
0.083 + 0.012
Back-
HRNet 38
0.596 1 0.046
0.133 1 0.015
0,086 1 0.010
ground
MST++ 39
0.628 + 0.050
0.107 + 0.010
0.076 = 0.005
NNos:: the srained model based on NTIRE dataset is defined take in the 3 channel imput, and cannot be applied to MSI data that has 4 channel imput.
Stectioll profil E200, 5001
Spectral proft (308, 5001
Spectraa prevle el (320, 200)
HSCNNN
Frdesioo
HRNet
8-009
fredition
MST+
SAN
0.3
Deaa
Figure 4: Re-training all three models with our Hyper-Skin dataset yielded notable enhancement in perfor- mance, particularly in the skin area. However, it worth noting that the first few bands of the reconstructed cube may not fully capture the variations present in the skin.
Se
21R
03
Speciral
Figure 5: The demonstration of a real selfie image captured with a smartphone shows that the trained model is capable of reconstructing spectral information at the skin location.
results demonstrate that the reconstruction of the skin area exhibits better performance, supporting the potential for low-cost hyperspectral skin solutions, such as reconstructing the facial skin-spectra cube from smartphone selfies.
4.2.2 Spectral Domain
Table shows SAM-based spectral evaluation results for data types: (RGB, VIS) and (MSI, NIR), with and without background. SAM ranges from 7/2 to 0, in which closer to indicating better reconstruction. After re- re-training models using our Hyper-Skin dataset, both data types showed significant performance improvements, as illustrated in Figure 4 Note that the performance of the models was generally lower when the background was present compared to when it was removed. This highlights the impact of background interference on the reconstruction results. These findings emphasize the potential of our dataset in enhancing hyperspectral reconstruction for applications related to skin analysis and other fields that require accurate spatial information. Note that the re-trained models also showed good performance on the (MSI, NIR) data. We further verify the reconstruction performance on the real RGB image taken by a smartphone. As shown in Figure 5 the reconstruction model capable of estimating the skin spectral information. Due to page constraints, the visualizations of these results provided as supplementary materials.

---

Limitations, Ethical Considerations and Societal Impact
We worked closely with the university's ethical review board to ensure the inclusivity, privacy, and ethical integrity of the Hyper-Skin dataset. This ensured responsible data usage for skin analysis research. Our data collection followed strict participant consent procedures, including providing participants with detailed project information before obtaining written and verbal consent. The ethics review protocol and consent procedures are provided as supplementary materials.
Limitations Our dataset's representativeness might be limited, potentially not capturing the full diversity of skin types, tones, and conditions across various populations. This could introduce biases and hinder model generalization. Additionally, while our deep neural network-based approach leverages the dataset for latent priors, it's constrained by dataset limitations. Novel cosmetology conditions not covered training might affect model performance due to distribution shifts, a common challenge in machine learning [4 45 particularly in medical datasets with skewed distributions To address this, we' 've secured ethical approval to expand data collection, targeting diverse participants and cosmetology conditions to enhance practical utility.
46 47..
Ethical Considerations To ensure ethical compliance throughout the data collection process, we implemented measures anonymize personally identifiable information, such assigning subject ID instead of using participants' real information. Robust security measures were also put in place to protect sensitive data from unauthorized access or misuse. We strictly followed guidelines provided by the research ethics board and obtained informed consent from every participant, respecting their autonomy and ensuring they understood how their data would be used.
Societal Impact Our Hyper- Skin dataset revolutionizes skin analysis by providing affordable and accessible solutions. With its ability to reconstruct skin spectral properties and estimate parameters like melanin and hemoglobin concentration, it empowers researchers and practitioners to develop low-cost skin analysis solutions directly for consumers. The dataset's societal impact extends to individuals monitoring their skin's well-being and skincare companies developing personalized products and innovative Al models. By driving advancements in skin analysis, the Hyper-Skin dataset benefits individuals, professionals, and the skincare industry as a whole.
Conclusion
This paper contributes the field of hyperspectral skin analysis by providing a comprehensive collection of facial skin hyperspectral data, named Hyper-Skin dataset. The novelty of this dataset lies in its spectral coverage in the VIS and NIR spectrum, offering potential applications in skin monitoring and customized cosmetic products at the consumer's 's fingertips. It serves as a valuable resource for algorithm development and evaluation, with future directions including dataset diversifi- cation, advanced analysis techniques, and interdisciplinary collaborations, inviting researchers and practitioners contribute to the advancement of hyperspectral skin analysis for human well-being.
References
[1] F. Vasefi, N. MacKinnon, R. B. Saager, A. J. Durkin, R. Chave, E. H. Lindsley, and D. Farkas, "Polarization-sensiivee hyperspectral imaging in vivo: multimode dermoscope for skin analysis," " Scientific reports, vol. 4, no. 1, pp. 1-10, 2014.
[2] L. Gevaux, J. Gierschendorf, J. Rengot, M. Cherel, P. Séroul, A. Nkengne, J. Robic, A. Trémeau, and M. Hébert, "Real-time skin chromophore estimation from hyperspectral images using neural network." " Skin Research and Technology, vol. 27, no. 2, pp. 163-177, 2021.
[3] G. N. Stamatas, B. Z. Zmudzka, N. Kollias, and J. Z. Beer. "Non-invasive measurements of skin pigmentation situ," Pigment Cell Research, vol. 17, no. pp. 618-626, 2004.
[4] R. Koprowski, S. Wilezyáski, Z. Wróbel, and B. Blorsska- "Calibration and seg- mentation of skin areas in hyperspectral imaging for the needs of dermatology." Biomedical engineering online, vol. 13, no. I, pp. 1-19, 2014.
10

---

[5] Q. He and R. K. Wang, "Analysis of skin morphological features and real-time monitoring using snapshot hyperspectral imaging," Biomedical Optics Express, vol. 10, no. 11, pp. 5625-5638, 2019.
[6] V. Lindholm, A.. Raita-Hakola, L. Annala, M. Salmivuori, L. Jeskamen, H. Saari, S. Kosken- mies, S. Pitkänen, I. Pölönen, and K. Isoherranen, "Differentiating malignant from benign pigmented or non-pigmented skin tumours- a pilot study on 3d hyperspectral imaging of complex skin surfaces and convolutional neural networks," Journal of clinical medicine, vol. 11, no. 7, p. 1914, 2022.
[7] Q. He and R. Wang, "Hyperspectral imaging enabled by an unmodified smartphone for analyzing skin morphological features and monitoring hemodynamics," Biomedical optics express, vol. 11, no. 2. pp. 895 910, 2020.
[8] S. Kim, D. Cho, J. Kim, M. Kim, S. Youn, J. E. Jang, M. Je, H. Lee, B. Lee, and D. L. Farkas, "Smartphone--asee multispectral imaging: system development and potential for mobile skin diagnosis." Biomedicul optics express, vol. 7, no. 12, pp. 5294-5307, 2016.
[9] T. Kim, M. A. Visbal-Onufrak, R. L. Konger, and Y. L. Kim, "Data-driven imaging of tissue inflammation using rgb-based hyperspectral reconstruction toward personal monitoring of dermatologic health," Biomedical optics express, vol. 8, no. 11, pp. 5282-5296, 2017.
[10] Spigulis and I. Oshina, "Snapshot rgb mapping of skin melanin and hemoglobin." Journal of Biomedical Optics, vol. 20, no. 5, pp. 503-050 503, 2015.
[11] D. Jakovels, U. Rubins, and J. Spigulis, "Rgb imaging system for mapping and monitoring of hemoglobin distribution in skin." in Imaging Spectrometry XVI, vol. 8158. SPIE, 2011, pp. 244-249.
[12] P. TTschandl, C. Rosendahl, and H. Kittler, "The 10000000 dataset, large collection of multi- source dermatoscopic images of common pigmented skin lesions," Scientific data, vol. 5, no. 1, pp. 1-9, 2018.
[13] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, C. Carrera, A. Barreiro, A. C. Halpem, and S. Puig, "Ben20000: Dermoscopic lesions in the wild," arXiv preprint arXiv: 1908. 02288, 2019.
[14] V. Rotemberg, N. Kurtansky, B. Stablein,,,,,, L. Caffery, E. Chousakos, N. Codella, M. Com- balia. S. Dusza, P. Guitera, and D. Gutman, "A petient-centrie dataset of images and metadata for identifying melanomas using clinical context," Scientific data, vol. 8, no. 1, p. 34, 2021.
[15] Y. Gu, Y.-P. Partridge, and J. Zhou, "A hyperspectral dermoscopy dataset for melanoma detec- tion," in OR 2.0 Cowtext-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Basse Procedures, and Skin Image Analysis: First International Workshop. OR 2.0 2018, 5th International Workshop. CARE 2018, 7th International Workshop, CLIP 2018, Third International Workshop, ISIC 2018, Held in Conjunction with MICCAI 2018, Granada, Spain, September 16 and 20, 2018, Proceedings 5. Springer, 2018, pp. 268 276.
[16] M. S. Chin, B. B. Freniere, Y.-C. Lo, H. Saleeby, S. Baker, H. M. Strom, R. A. Ignotz, J. F. Lalikos, and J. Fitzgerald, "Hyperspectral imaging for early detection of oxygenation and perfusion changes irradiated skin," Journal of Biomedical Optics, vol. 17, no. 2, pp. 026 010 0110026000 010, 2012.
[17] F. Manni, F. van der Sommen, S. Zinger, C. Shan, R. Holthuizen, M. Lai, G. Buström, R. J. Hoveling, E. Edström, and A. Elmi-Terander, "Hyperspectral imaging for skin feature detection: Advances in markerless tracking for spine surgery." Applied Sciences, vol. 10, no. 12, 4078, 2020.
[18] A. Shahzad, M. N. Saad, N. Walter, A. S. Malik, and F. Meriaudeau, "Hyperspectral venous image quality assessment for optimum illumination range selection based on skin tone charac- teristics." Biomedical engineering online, vol. 13, no. 1, pp. 1-13, 2014.

---

[19] T. F. Chen and G. V. Baranoski, "Effective compression and reconstruction of human skin hyperspectral reflectance databases," in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IERE, 2015, pp. 7027-7030.
[20] A. Nkengne, J. Robic, P. Seroul, S. Gueheunneux, M. Jomier, and K. Vie, "Spectracam@: A new polarized hyperspectral imaging system for repeatable and reproducible in vivo skin quan- tification of melanin, total hemoglobin, and oxygen saturation," Skin Research and Technology. vol. 24, no. 1, pp. 99-107, 2018.
[21] L. Gevaux, J. L. Perrot, and M. Hébert, "Multispectral and hyperspectral imaging for skin acquisition and analysis." Technology in Practical Dermatology: Non-Invasive Imaging, Lasers and Ulcer Management, pp. 271-279, 2020.
[22] Benavides, T. M. Oberyszyn, A. M. VanBuskirk, V. E. Reeve, and D. F. Kusewitt, "The hairless mouse in skin research," " Journal of dermatological science, vol. 53, no. 1, pp. 10-18, 2009.
[23] Q Xu, Z. Liu, Z. Cao, Y. Shi. N. Yang, G. Cao. C. Zhang, R. Sun, and C. Zhang, "Topical astilbin ameliorates imiquimod -nduuce psoriasis- like skin lesions in skh-1 mice via suppression dendritic cell-th17 inflammation axis." Journal of Cellular and Molecular Medicine, vol. 26, no. 4, pp. 1292, 2022.
[24] D. I. F. Yasuma, T. Mitsunaga and S. Nayar, "Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum," " IEEE Transactions on Image Processing. vol. 99, Mar 2010.
[25] A. Chakrabarti and T. Zickler, "Statistics of real-world hyperspectral images," in CVPR 2011. IEEE, 2011, pp. 193-200.
[26] I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, "High-quality hyperspectral reconstruction using a spectral prior," ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017), vol. 36, no. 6, pp. 218:1-13, 2017.
[27] S.-H. Baek, H. Ikoma, D. S. Jeon, Y. Li, W. Heidrich, G. Wetzstein, and M. H. Kim, "Single- shot hyperspectral-dephe imaging with learned diffractive optics," in Proc. IEFE International Conference on Computer Vision (ICCV), October 2021.
[28] B. Arad, O. Ben-Shahar, R. Timofte, L. Van Gool, L. Zhang, M.-H. Yang. Z. Xiong, C. Chen,
Z. Shi, D. Liu, F. Wu, C. Lanaras, S. Galliani, K. Schindler, T. Stiebel, S. Koppers, P. Seltsam,
R. Zhou, M. EI Helou, F. Lahoud, M. Shahpaski, K. Zheng, L.. Gao, B. Zhang, X. Cui, H. Yu,
Y. B. Can, A. Alvarez-Gila, J. van de Weijer, F. Garrote, A. Galdran, M. Sharma, S. Koundinya,
A. Upadhyyay, R. Manekar, R. Mukhopadhyay, H. Sharma, S. Chaudhury, K. Nagasubramanian, and S. Sarkar, "Ntire 2018 challenge on spectral reconstruction from rgb images." in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2018, pp. 1042-104 209.
S. Ghosal, A. K. Singh, A. Singh, B. Ganapathysubramannin
[29] B. Arad, R. Timofte, O. Ben-Shahar, Y.-T. Lin, G. Finlayson, S. Givati, J. Li, C. Wu, R. Song, Y. Li, F. Liu, Z. Lang, W. Wei, L. Zhang, J. Nic. Y. Zhao, L.-M. Po, Q. Yan, W. Liu, T. Lin. Y. Kim, C. Shin, K. Rho, S. Kim, Z. ZHU, J. HOU, H. Sun, J. Ren, Z. Fang. Y. Yan, H. Peng, X. Chen, J. Zhao, T. Stiebel, S. Koppers, D. Merhof, H. Gupta, K. Mitra, B. J. Fubara, M. Sedky, D. Dyke, A. Banerjee, A. Pairecha, S. sabarinathan, K. Uma, D. S. Vinothini, B. Sathya Bama, and S. M. Md Mansoor Roomi, "Ntire 2020 challenge on spectral reconstruction from an rgb image," in 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020. pp. 1802-1822
[30]
B. Arad, Timofte, R. Yahel, N. Morag, A. Bernat, Y. Cai, J. Lin, Z. Lin, H. Wang, Y. Zhang, H. Pfister, L. Van Gool, S. Liu, Y. Li, C. Feng, L. Lei, J. Li, S. Du, C. Wu, Y. Leng, R. Song,
M. Zhang, C. Song, S. Zhao, Z. Lang, W. Wei, L. Zhang, R. Dian, T. Shan, A. Guo, C. Feng,
J. Liu, M. Agarla, S. Bianco, M. Buzzelli, L. Celona, R. Schettini, J. He, Y. Xiao, J. Xiao, Q. Yuan, J. Li, L. Zhang, T. Kwon, D. Ryu, H. Bae, H.-H. Yang, H.-E Chang, Z.-K. Huang, W.- T. Chen, S.-Y. Kuo, J. Chen, H. Li, S. Liu, S. Sabarinathan, K. Uma, B. S. Bama, and S. M. M. Roomi, "Ntire 2022 spectral recovery challenge and data set," 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022, pp. 862-880.
12

---

[31] P. C. Ng. Y. Verdie, J. Lu. and K. N. Plataniotis, "Efficient hyperspectral reconstruction from rgb images with line- lie-pixel deconvolution," " in 2022 IEEE 14th Image, Video, and Multidimensional Signal Processing Workshop (IVMSP), 2022, pp. 1-5.
[32] L. J. Denes, P. Metes, and Y. Liu, Hypeerspectrl face database. Carnegie Mellon University, The Robotics Institute, 2002.
[33] M. Magnusson, J. Sigurdsson, S. E. Armansson, M. O. Ulfarsson, H. Deborah, and J. R. Sveinsson, "Creating rgb images from hyperspectral images using color matching function," in IGARSS 2020 2020 IEEE International Geoscience and Remote Sensing Symposium, 2020, pp. 1045-2048.
[34] J. Jiang, D. Liu, J. Gu, and S. Süsstrunk, "What is the space of spectral sensitivity functions for digital color cameras?" in 2013 IEEE Workshop on Applications of Computer Vision (WACV), 2013, pp. 168-179.
[35] R. Kawakami, Z. Hongxun, T. Tan, and K. Ikeuchi, "Camera spectral sensitivity and white balance estimation from sky images," International Journal of Computer Vision, June 2013.
[36] J. Zhang, R. Su, Q. Fu, W. Ren, F. Heide, and Y. Nie, "A survey on computational spectral reconstruction methods from rgb to hyperspectral imaging." " Scientific reports, vol. 12, no. I, pp. 1-17, 2022.
[37] Z. Shi, C. Chen, Z. Xiong, D. Liu, and F. Wu, "Hsenn+: Advanced cnn-based hyperspectral recovery from rgb images," Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition Workshops, 2018, pp. 939-947.
[38] Y. Zhao, L.-M. Po, Q. Yan, W. Liu, and T. Lin, "Hierarchical regression network for spectral reconstruction from rgb images," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020. pp. 422 423.
[39] Y. Cai, J. Lin, Z. Lin, H. Wang, Y. Zhang, H. Pfister, R. Timofte, and L. Van Gool, "Mst+++: statt spectral-wise transformer for efficient spectral reconstruction," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 745-755.
[40] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612, 2004.
[41] A. Kruse, A. Lefkoff, Boardman, K. Heidebrecht, A. Shapiro, Barloon, and A. Goetz, "The spectral image processing system (sips) intercivte visualization and analysis of imaging spectrometer data," Remote sensing of environment, vol. 44, no. 2-3, pp. 145-163, 1993.
[42] Y. Cai, "A toolbox for spectral compressive imaging." " [Online]. Available:
https
/ggihub.coo//aiyuannaa 998/MMST
[43] A. Boaz, "Gnu general public license version 3." [Online]. Available: https://github.com/ bouzarad//NTIRE2222 hatpo//sssccctrrbboooanniiiilCC.
[44] Y. Ovadia, E. Fertig, J. Ren, 2. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshninarayanan, and J. Snoek, "Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift," " Advances in neural information processing systems, vol. 32, 2019.
[45] T. Fang. N. Lu, G. Niu, and M. Sugiyama, "Rethinking importance weighting for deep learning under distribution shift," " Advances in neural information processing systems, vol. 33, pp. 11 9999991222 007, 2020.
[46] C. J. Kelly, A. Karthikesalingam, M. Suleyman, G. Corrado, and D. King, "Key challenges for delivering clinical impact with artificial intelligence," BMC medicine, vol. 17, pp. 1-9, 2019.
[47] L. Alzubaidi, J. Bai, A. Al-Sabaawi, Santamaría, A. Albahri, B. S. N. Al-dabbagh, M. A. Fadhel, M. Manoufali, J. Zhang, and A. H. Al-Timemy, "A survey on deep learning tools dealing with data searcity: definitions, challenges, solutions, tips, and applications," Journal of Big Data, vol. 10, no. 1, p. 46, 2023.
13


Reviews:
[{"review_id": "qepNwHTAHB", "reviewer": "Reviewer_cCX8", "summary": "", "questions": "", "limitations": "The authors adequately addressed the limitations of the work and its potential negative social impact in the text and appendix.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The strengths of this paper are (1) that it secured hyperspectral data from 400 nm to 1,000 nm, and (2) that it photographed the faces and expressions of real applicants from various angles.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "mUQSqZ3BVx", "reviewer": "Reviewer_rZAx", "summary": "", "questions": "", "limitations": "- As mentioned by the authors, the representativeness of the dataset is a potential limitation, as it may not fully encompass the diverse range of skin types, tones, and conditions present in different populations. This can introduce biases and hinder the generalizability of models trained on the dataset.  \n - While the dataset includes 330 hyperspectral cubes, the number of subjects (51) is relatively small (with 4 participants chosen randomly for test set), which may limit the generalizability of the findings. ", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "- Provision of publicly accessible high spatial and spectral resolution dataset for hyperspectral skin analysis \n- Precise and clear dataset curation process  ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "VMwL7s86AY", "reviewer": "Reviewer_5oGQ", "summary": "", "questions": "", "limitations": "They clearly address the limitations of their work and provide future plans to cover the limitations.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. RGB data construction: Section 3.1 provides a clear and comprehensive explanation of the data collection methods and the procedure used for generating RGB/MSI data. The authors' detailed description allows readers to understand the data acquisition process thoroughly, ensuring transparency and reproducibility of their research.\n2. Increased resolution and dataset size: One of the key strengths of their proposed dataset is its higher resolution and larger quantity compared to previous hyperspectral face datasets. This enhancement is a significant contribution to the field, as it allows for more detailed and comprehensive analysis of facial spectral information, offering researchers and practitioners a valuable resource for their studies.\n3. Novel method for RGB/MSI data reconstruction: The approach used by the authors to reconstruct RGB/MSI data from HSI data is captivating and holds promise for practical applications. The decoupling of HSI data into (RGB, VIS) and (MSI, NIR) is an innovative concept that provides insights into the spectral characteristics of different facial features. This novel technique has the potential to advance research in the domain of hyperspectral imaging, opening up possibilities for various applications beyond facial analysis.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "dFAFyiGA0j", "reviewer": "Reviewer_CmSs", "summary": "", "questions": "", "limitations": "The paper mentions the limitations of this work, which is good. The main limitation is that it cannot cover all types and conditions of skin. The author states that additional efforts will be taken to collect more samples from different ages, races, and regions.", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The advanced aspects of this work are mainly reflected in the following areas:\n\n1. It is a hyperspectral dataset specifically for facial skin spectral reconstruction, not a general dataset for natural scenes or everyday objects.\n\n2. It is the first facial skin hyperspectral dataset that includes both visible light and near-infrared spectra, providing comprehensive information for studying surface and deep features of the skin.\n\n3. It provides a facial skin hyperspectral dataset of synthetic RGB images and multispectral images generated by real camera response functions, promoting high-spectrum skin analysis on consumer-grade devices.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Submission: Hyper-Skin (**#429**)
Bottom line: **accept (as poster)**

**Summary**: The authors collected novel data using a thoroughly described and in-house developed methodology: 306 images representing 3 viewing angle captures of human faces with two facial expressions were collected using a multi-spectral line-scanning camera. The final dataset is made available to the community as 1024x1024 pixel (original resolution) by 31-bands (downsampled) images, together with a number of assets that will allow a number of potential applications.

Pros: extending beyond existing work by (1) larger number of bands (and wider range of covered wavelengths), paired with (2) high-resolution standardized capture of human faces, as well as (3) benchmarking work on present dataset, (4) together with auxiliary assets (camera response functions, etc.); see particularly Table 1.

Cons: primarily (1) the small number of participants (selection bias), and (2) missing annotations, which makes it unlikely to support calibrating AI models for exploring specific applications (e.g., skin disease detection). In addition, (3) the exact applications are still somewhat vague (see however below).

Decision: As Area Chair, I am satisfied with the responses to the reviews, and I believe that the community will have to explore the dataset (as provided, and on request also the original 448-bands captures) to **discover applications** that the authors (and reviewers) probably cannot foresee. Given the primary nature of the data presented, I also believe that the data will find its use cases, and so support accepting this as a poster for NeurIPS 2023.

----

Reviewer cCX8: biggest downside mentioned was missing/unclear link between data and use cases; authors acknowledged and clarified in the manuscript -- at the same time, I (Area Chair) take it that applications will often require to "play with data" that is available, and so believe that with the revisions, the authors have provided sufficient motivation/incentive for the community to do exactly that.

----

Reviewer rZAx: biggest downside mentioned was/is lack of of skin type representation together with small sample size; authors acknowledge and then (unfortunately) **do not provide specific rebuttals**. It is my (AC, with now several years working as a data analyst in a dermatology department) belief that given the difficulty of this topic (racial diversity, representation of skin types), it would have been very difficult for the authors to provide a *scientifically and politically acceptable* argument that would not have required them to increase the dataset size by at least a factor of 2 or 3 (if not an order of magnitude). As such, I agree with the reviewer (broadly speaking), but yet believe that it is beneficial to publish the dataset **as is** for the benefit of the community; if the SAC/PCs would like to request that the authors add more language (beyond the second paragraph in the limitations section as it currently stands), I would support that, but do not believe this to be an impediment to publication -- with such a limitation clearly expressed! It is equally unfortunate that this reviewer did not provide a follow-up to the authors' (somewhat inconsequential) rebuttal, depriving me of any strong potential arguments against accepting the dataset **for this reason** at least...

----

Reviewer 5oGQ: biggest downside mentioned was/is the potential lack of novelty; I believe that while prior datasets each address one of the parameters the authors bring together (resolution, number of bands, image subject, etc.), the present dataset does, indeed, **bring these factors neatly together** in a single dataset (with the exception of being as large as a prior dataset with >200 subjects). All other concerns seem to be satisfactorily addressed.

----

Reviewer CmSs: highest rating (within the set of reviewers), and pretty much all (minor) concerns addressed.

----

AC concerns: the authors have addressed all concerns **to my satisfaction** (some concerns are limitations, which are equally addressed to my satisfaction).
Decision: Accept (Poster)