SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pai Chet Ng<sup>1</sup>, Zhixiang Chi<sup>1</sup>, Yannick Verdie<sup>2</sup>, Juwei Lu<sup>2</sup>, Konstantinos N. Plataniotis<sup>1</sup>  
<sup>1</sup> The Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto  
<sup>2</sup> Huawei Noah's Ark Laboratory, Huawei Canada
We introduce Hyper-Skin, a hyperspectral dataset covering wide range of wavelengths from visible (VIS) spectrum  $(400\mathrm{nm} - 700\mathrm{nm})$  to near-infrared (NIR) spectrum  $(700\mathrm{nm} - 1000\mathrm{nm})$ , uniquely designed to facilitate research on facial skin-spectra reconstruction. By reconstructing skin spectra from RGB images, our dataset enables the study of hyperspectral skin analysis, such as melanin and hemoglobin concentrations, directly on the consumer device. Overcoming limitations of existing datasets, Hyper-Skin consists of diverse facial skin data collected with a pushbroom hyperspectral camera. With 330 hyperspectral cubes from 51 subjects, the dataset covers the facial skin from different angles and facial poses. Each hyperspectral cube has dimensions of  $1024\times 1024\times 448$ , resulting in millions of spectra vectors per image. The dataset, carefully curated in adherence to ethical guidelines, includes paired hyperspectral images and synthetic RGB images generated using real camera responses. We demonstrate the efficacy of our dataset by showcasing skin spectra reconstruction using state-of-the-art models on 31 bands of hyperspectral data resampled in the VIS and NIR spectrum. This Hyper-Skin dataset would be a valuable resource to NeurIPS community, encouraging the development of novel algorithms for skin spectral reconstruction while fostering interdisciplinary collaboration in hyperspectral skin analysis related to cosmetology and skin's well-being. Instructions to request the data and the related benchmarking codes are publicly available at: https://github.com/hyperspectral-skin/Hyper-Skin-2023.
Hyperspectral imaging offers a comprehensive and non-invasive approach for facial skin analysis, capturing detailed spatio-spectral information across a wide range of wavelengths [1, 2]. This three-dimensional hyperspectral cube surpasses the limitations of single-point measurements, providing a deeper understanding of facial skin characteristics and spatial distribution [3]. Previous studies have demonstrated the potential of hyperspectral skin analysis in dermatology [4], cosmetics [5], and skin's well-being [6], paving the way for advanced analysis and applications in these domains. This paper introduces "hyper-skin", a hyperspectral skin dataset uniquely designed to facilitate the development of algorithms targeting on consumer-based cosmetology applications. This unique dataset is curated with this specific goal in mind, focusing its practical relevance within the consumer-based cosmetology and skin beauty.
Despite the potential of hyperspectral skin analysis on cosmetology and skin beauty, the high cost and limited accessibility of hyperspectral imaging systems have limited their widespread adoption. Consumer cameras, particularly those embedded in smartphones, have become an integral part of daily life and are extensively used for capturing selfies and everyday images. Hence, many works study the use of RGB images from consumer cameras for skin analysis [7, 8, 9]. While RGB
Figure 1: A glimpse of our Hyper-Skin dataset, covering the skin spectra in the visible spectrum (400nm - 700nm) and near-infrared spectrum (700nm - 1000nm).
images have been used for certain skin analysis tasks [10, 11], they lack the ability to capture the comprehensive spatio-spectral information provided by hyperspectral imaging, limiting the depth of skin analysis. In light of the prevalence of consumer cameras, an intriguing idea emerges: Can we reconstruct valuable information from expensive hyperspectral cubes using accessible RGB images, enabling hyperspectral skin analysis directly on consumer devices?
This highlights the need for a comprehensive dataset to develop computational reconstruction methods for the question above. While RGB datasets such as those from the International Skin Imaging Collaboration (ISIC) competition series (2016 - 2020) capture visual information, they lack the corresponding hyperspectral data required for studying hyperspectral reconstruction [12, 13, 14]. On the other hand, hyperspectral datasets enable the exploration of relationships between skin spectra and spatial distribution. Although the RGB counterpart can be synthetically generated from a given hyperspectral cube using a known camera response function, publicly available hyperspectral datasets focusing specifically on facial skin analysis are limited and often inaccessible to the public. Furthermore, existing hyperspectral datasets primarily focus on the visible (VIS) spectrum (400nm - 700nm), disregarding the valuable near-infrared (NIR) spectrum (700nm - 1000nm). These limitations highlight the necessity for a hyperspectral dataset that addresses these gaps and facilitates the development of low-cost and accessible hyperspectral skin analysis on consumer devices.
Our Contributions Our Hyper-Skin dataset is uniquely designed to unlock the potential of hyperspectral skin analysis directly on the consumer device. With high spatial and spectral resolution, i.e.,  $1024 \times 1024 \times 448$ , Hyper-Skin offers an extensive collection of hyperspectral cubes, yielding over a million spectra per image. Notably, we offer synthetic RGB images synthesized from 28 real camera response functions, allowing for versatile experimental setups. What sets Hyper-Skin apart is its comprehensive spectral coverage, including both the VIS and NIR spectrum, facilitating a holistic understanding of various aspects of human facial skin, enabling new possibilities for consumer applications to see beyond the visual appearance of their selfies and gain valuable insights into their skin's physiological characteristics, such as melanin and hemoglobin concentrations.
The potential hyperspectral solutions in the skin-related analysis have encouraged the curation of hyperspectral datasets. This section reviews existing hyperspectral datasets related to skin analysis, as summarized in Table 2, and reconstruction aiming to provide affordable hyperspectral solutions accessible to consumers.
Table 1: Comparison between Skin-related Data
Work	Experimental Subjects	Number of Subjects	Spectral Range	Spectral Resolution	Spatial Resolution	Acquisition Device
Ours	Full face	51	400 - 1000 nm	1.34 nm	1024 x 1024	Specim FX10
[1]	Arm	2	467 - 857 nm	13 nm	NA	SkinSpect dermoscope
[2]	Facial skin	204	400 - 700 nm	3.3 nm	1148 x 948	SpectraCam®
[15]	Skin lesions	330*	465 - 630 nm	10.3125 nm	512 x 272	Ximea MQ022HG-IM-SM4X4-VIS
[16]	Hairless mice	10	500 - 660 nm	NA	NA	OxyVuTM -2, HyperMed T
[17]	Spine area	17	450 - 950 nm	12.2 nm	500 x 250	snapshot hyperspectral imaging (HSI) camera
[18]	Fore arm	80	380 - 1055 nm	2.8 nm	450 x 1310	Specim® Spectral Camera PS V10E
[4]	Hand	45**	397 - 1030 nm	0.79 nm	899 x 1312	PFD-V10E line-scan camera
[19]	Skin reflectance	144***	250 - 2500 nm	5 nm	NA	NA
[9]	Mouse	100**	500 - 1000 nm	5 nm	640 x 480	custom-built LCTF
* Dermoscopy images, ** Reflectance images, *** Synthetic reflectance
Skin-related Datasets SpectraCam and SpectraFace hyperspectral cameras [20] has been used to collect the data of normal and pathological skin with a spectral resolution of 31 wavelengths in the VIS spectrum [21]. The hyperspectral dermoscopy dataset consists of 330 images, including 80 melanoma images, 180 dysplastic nevus images, and 70 images of other skin lesions, with a spatial resolution of  $512 \times 272$  pixels and 16 spectral bands ranging from  $465\mathrm{nm}$  to  $630\mathrm{nm}$  [15]. Hyperspectral dataset of 20 nude mice was collected by [16] as an alternative to human skin [22] to study acute changes in oxygenation and perfusion in irradiated skin. For markerless tracking in spinal navigation, [17] captured hyperspectral images of the skin from 17 healthy volunteers, with a spatial resolution of  $1080 \times 2048$  and 41 spectral bands in the VISt to NIR range (450-950nm). Both work in [18] and [4] used the same Specim Spectral Camera PS V10E to acquire hyperspectral data covering the visible to NIR range (380-1055nm) with 1040 bands and a spatial resolution of  $450 \times 1310$ . The former dataset by [18], containing data from 80 subjects, focused on vein localization, whereas [4] used the dataset for a routine dermatological examinations. While references [18, 4, 17] involve capturing NIR spectral information, it's important to note that these datasets are not publicly accessible. Despite the potential of hyperspectral imaging in skin-related applications, most of these existing datasets relied on expensive imaging systems with a primary focus on scientific applications, and no efforts were made to provide low-cost hyperspectral solutions for consumer devices.
Skin Spectral Reconstruction Datasets Although datasets specifically focused on skin spectra reconstruction are limited, there have been notable contributions in this field. One such dataset is the Skin Hyperspectral Reflectance Database (SHRD) [19], which includes 144 skin directional-hemispherical curves obtained through a novel hyperspectral light transport model. This dataset provides valuable insights into reconstructing hyperspectral information on human skin. Another study explores the reconstruction of hyperspectral information in mice skin [9], using 26 SKH-1 hairless albino mice as a model system [23]. The researchers propose a mathematical approach to reconstruct hyperspectral data from RGB images, allowing visualization of hemoglobin content across a large skin area without the need for expensive hyperspectral imaging systems. However, these skin-spectral reconstruction datasets have limitations in terms of sample size and spatial resolution, particularly in their coverage of facial images from various angles and poses. Consequently, progress in skin-spectral reconstruction has been relatively slower compared to hyperspectral reconstruction on natural scenes or everyday objects.
Natural Scenes and Everyday Objects Reconstruction Datasets Compared to proprietary and unavailable skin-related datasets, several hyperspectral datasets on natural scenes and everyday objects have been made publicly available for the study of hyperspectral reconstruction. The CAVE dataset consists of 32 scenes with a spatial resolution of  $512 \times 512$  pixels, 31 spectral bands ranging from  $400\mathrm{nm}$  to  $700\mathrm{nm}$  at  $10\mathrm{nm}$  intervals [24]. The HARVARD dataset includes 50 images captured under daylight illumination, with 31 spectral bands spanning from  $420\mathrm{nm}$  to  $720\mathrm{nm}$ , and a spatial resolution of  $464\times 346$  pixels [25]. The KAIST dataset comprises 30 hyperspectral images with a spatial resolution of  $3376 \times 2704$  pixels and a spectral range from  $420\mathrm{nm}$  to  $720\mathrm{nm}$  [26]. The
KAIST-depth dataset features 16 indoor scenes with a spectral range of  $420\mathrm{nm}$  to  $680\mathrm{nm}$ , 27 spectral channels, and a spatial resolution of  $2824 \times 4240$  pixels [27]. The New Trends in Image Restoration and Enhancement (NTIRE) series of datasets, including NTIRE2018, NTIRE2020, and NTIRE2022, have significantly advanced spectral reconstruction from RGB images, with varying sizes, spectral resolutions, and spectral ranges [28, 29, 30, 31]. Despite primarily focusing on natural scenes and generic objects, these publicly available datasets offer valuable resources, including facilitating the development of pre-trained models, that can be extended to skin spectral reconstruction tasks.
3 Hyper-Skin Data Curation and Preparation
This section outlines the methodology we employed for collecting facial skin data and provides a detailed description of our carefully curated Hyper-Skin dataset.
Figure 2: The image on the right illustrates our experimental setup for data collection, while the accompanying schematic representation provides a visual depiction of the setup.
The data collection process was conducted carefully, taking into account the setup of devices and recruitment of participants while ensuring adherence to the university's research ethics protocol. We successfully recruited 51 participants who contributed a total of 306 hyperspectral data. To maintain the privacy and sensitivity of the human subjects involved, we have implemented a credentialization procedure to access the dataset. Interested users will be required to digitally sign an End User License Agreement (EULA) online, which outlines the terms and conditions for using the dataset, including provisions for using only the authorized image in future publications. Detailed instructions for requesting the dataset will be publicly available in our GitHub repository, where users can find a digital EULA form to facilitate the data access request. Once the EULA form is signed and submitted, users will receive a secure link via email to download the data within 24 hours.
Data Acquisition Devices and Set Up The Hyper-Skin dataset was obtained using a Specim FX10 camera, covering 448 spectral bands from  $400\mathrm{nm}$  to  $1000\mathrm{nm}$ . Consider multiple factors, including participant safety, image quality, and spectral resolution, we opted to use a pushbroom camera rather than the Liquid Crystal Tunable Filter (LCTF) system used by [32]. The camera was moved using a customized scanner for precise scanning, as shown in Figure 1. The distance between the camera and the face was set at  $40\mathrm{cm}$ , providing a spatial resolution of  $1024\times 1024$  pixels. The scanner and camera were controlled by a computer running LUMO recorder software. Further setup details are available in the supplementary material. With a frame rate of  $45\mathrm{Hz}$  for one line, it took approximately 22.7 seconds to capture all 1024 lines. To minimize artifacts from line scanning, participants used a chin rest for stability. Halogen lamps illuminated the scene across the visible to near-infrared spectrum. Since ensuring participant safety (particularly for the eyes area) was a top priority, particularly for their eyes, the illumination level of the halogen lamps is carefully adjusted following the manufacturer's advice to prevent any risk to participants' eyes. Using the Specim FX10 camera resulted in high-quality images, setting our dataset apart from the mentioned dataset [32], which contains noisy and blurry images that affect skin texture visibility. Differing from the CMU dataset [32], which features a  $10\mathrm{nm}$  step size for 65 bands, our dataset encompasses both VIS and NIR spectrum with finer resolution.
Table 2: Hyper-Skin data pairs
Description	(RGB, VIS)	(MSI, NIR)
Input	RGB	MSI (RGB + Infrared at 960nm)
Output	VIS (400nm - 700nm)	NIR (700nm - 1000nm)
Skin physiological features	surface-level characteristics (e.g., pig-mentation and melanin map)	deeper tissue properties (e.g., collagen content and hemoglobin map)
Data Acquisition Process Participants were recruited through online forums and email advertisements, and their participation involved signing an informed consent form in accordance with the human research ethics protocol. The approved ethics protocol can be found in the supplementary materials. During the data acquisition process, participants were seated on a stool and asked to rest their face on a chin rest while maintaining stillness. Initially, participants were instructed to have a neutral facial expression, and three face images were captured from different viewpoints (front, left, and right) by rotating the chin rest. This process was then repeated with participants instructed to smile. A total of six images were collected for each participant. Throughout the camera scanning, a halogen light remained on. It's worth noting that even with minimal participant movement, slight shifting may occur as the FX10 camera scans line by line. To ensure high-quality images, the captured images were manually inspected by the investigator, and if any shifting was observed, the image was retaken until satisfactory results were achieved. Throughout the entire process, participant anonymity and confidentiality were strictly maintained.
Participants Demographic and Cosmetology Condition Our data collection campaign attracted 51 participants, most participants are in their early 20s and 30s, with a smaller representation from other age groups (10s, 40s-50s). Male participants slightly outnumbered females, potentially due to the gender distribution in the Department of Electrical and Computer Engineering. The majority of participants identified as Asian, with a smaller number identifying as European or Latino. To improve the generalizability of our findings, we have applied for an extension of our research ethics protocol to conduct another data collection next year, aiming to include a more diverse sample.
3.2 Data Preparation
The Hyper-Skin dataset was created by collecting RAW hyperspectral data, which were then radiometrically calibrated and resampled into two separate 31-band datasets. One dataset covers the visible spectrum from  $400\mathrm{nm}$  to  $700\mathrm{nm}$ , while the other dataset covers the near-infrared spectrum from  $700\mathrm{nm}$  to  $1000\mathrm{nm}$ . Additionally, synthetic RGB and Multispectral (MSI) data were generated, including RGB images and an infrared image at  $960\mathrm{nm}$ . The Hyper-Skin dataset consists of two types of data: (RGB, VIS) and (MSI, NIR), offering different skin analysis capabilities. The visible spectrum data allows for the analysis of surface-level skin characteristics, such as melanin concentration, blood oxygenation, pigmentation, and vascularization. On the other hand, the near-infrared spectrum data enables the study of deeper tissue properties, including water content, collagen content, subcutaneous blood vessels, and tissue oxygenation. As summarized in Table 2, by providing these two distinct ranges of hyperspectral data, the Hyper-Skin dataset caters to different needs in skin analysis and facilitates comprehensive investigations of various skin features.
Data Preprocessing We applied radiometric calibration on the RAW hyperspectral data to extract spectral reflectance information. This involved capturing a white reference image, representing a spectrally neutral surface with consistent reflectance values across all bands. A dark reference image was also obtained by closing the camera lens during capture. For precise calibration, selecting an appropriate white reference was crucial. After consultation with the camera vendor, we opted for cost-effective Teflon instead of Spectralon panels, as it provided satisfactory spectral response. The preprocessing steps included subtracting dark reference values to eliminate noise, and dividing by white reference values to normalize and convert data to reflectance values, yielding the desired spectral reflectance data.
RGB and MSI data Generation The raw hyperspectral cube with 448 bands was resampled into two sets of 31-band data using SciPy's interpolation function. This downsampling to 31 bands
is in line with existing practices in hyperspectral reconstruction studies, similar to the CAVE and NTIRE2018-2022 datasets. It strikes a balance between data richness and size. This approach retains hyperspectral differentiation and computational efficiency for analysis, making the data more accessible compared to the 448-band dataset, which exceeds 1TB in size. While the complete 448-band dataset is substantial in terms of size, we are prepared to provide it upon specific request. The availability of the 31-band data addresses data transfer constraints, while the backup 448-band data ensures comprehensive access to the dataset.
For realistic RGB data generation, we adopted the HSI2RGB simulation pipeline based on ideal color-matching functions as outlined in [33]. Our emulation of consumer camera-captured images incorporates 28 camera response functions from [34] and [35], encompassing various cameras like DSLR and smartphones. Further details on the measurement setup and gathering camera spectral sensitivity information can be found in [34]. While we do possess actual RGB data captured by smartphone sensors, their current utility is constrained by concerns related to alignment quality issues stem from variations in camera models, viewing angles, and overlapping fields of view. In pursuit of maintaining high-quality aligned pairs, we have chosen to provide synthetic RGB images that are perfectly aligned with their hyperspectral counterparts. This approach aligns with existing practices such as those in the NTIRE2018-2022 challenges.
Data Description We intentionally chose 4 participants from the total of 51 to form the testing data. Each participant contributed 6 images, covering 2 facial expressions and 3 face poses. This selection ensured a comprehensive representation of facial poses and expressions in the testing dataset. The choice of these 4 participants was deliberate, based on their explicit consent for image use in publications, adhering to ethical standards. The remaining data was exclusively used for offline training. Collecting diverse participant images encompassed a variety of natural facial poses and expressions observed in selfies. Opting for a participant-specific approach, rather than random partitioning, prevents data overlap within the same participant in the training set, reducing potential bias. This strategic participant selection safeguards dataset integrity and subsequent analysis.
4 Evaluation and Benchmarks
This section discusses the benchmark design with the facial skin-spectra reconstruction and then presents the experimental results from both the spatial and spectral domains.
4.1 Facial Skin-spectra Reconstruction
The facial skin-spectra reconstruction task focuses on reconstructing the hyperspectral cube of the facial skin using the provided RGB image. Given a pair of RGB data represented as  $R \in \mathbb{R}^{w \times h \times c}$  and the hyperspectral cube denoted as  $H \in \mathbb{R}^{w \times h \times C}$ , where  $c < C$ , the objective of the reconstruction task can be formulated as follows:
$$
H = f (R; \Theta), \tag {1}
$$
where the goal is to find the function  $f(\cdot ;\Theta)$  parameterized by  $\Theta$  that maps the RGB data  $R$  to the hyperspectral cube  $H$ . Given the extensive research on hyperspectral reconstruction for natural scenes or everyday objects, we can leverage existing hyperspectral reconstruction models as baseline models for our specific facial skin-spectra reconstruction problem.
4.1.1 Baseline Models
Numerous methods have been developed to address hyperspectral reconstruction from RGB images. Interested readers can refer to the survey by [36] for a list of representative models that have been developed over the past two decades. For our benchmark design, we specifically consider three models, i.e., Hyperspectral Convolutional Neural Network (HSCNN) [37], Hierarchical Regression Network (HRNet) [38], and Multi-stage spectral-wise transformer (MST++) [39], that have emerged as winners in the NTIRE competition series held in conjunction with CVPR from the year of 2018, 2020 to 2022, respectively.
4.1.2 Evaluation Metrics
We consider two types of metrics: Structural Similarity Index (SSIM) for spatial evaluation and Spectral Angle Mapper (SAM) for spectral evaluation. Our emphasis lies in assessing the facial skin spectra reconstruction in human subjects, excluding the background image from analysis. This approach allows us to precisely gauge physiological properties like melanin and hemoglobin concentrations, essential for effective hyperspectral skin analysis.