# Coupled VAE and Interpolator approach for Fast Hyperspectral Image Emulation

Chedly Ben Azizi, Claire Guilloteau, Gilles Roussel, Matthieu Puigt

► To cite this version:

Chedly Ben Azizi, Claire Guilloteau, Gilles Roussel, Matthieu Puigt. Coupled VAE and Interpolator approach for Fast Hyperspectral Image Emulation. 14th Workshop on Hyperspectral Images and Signal Processing: Evolution in Remote Sensing - WHISPERS, Dec 2024, Helsinki, Finland. pp.1-5, 10.1109/WHISPERS65427.2024.10876458 . hal-04772033

HAL Id: hal-04772033

https://hal.science/hal-04772033v1

Submitted on 8 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# COUPLED VAE AND INTERPOLATOR APPROACH FOR FAST HYPERSPECTRAL IMAGE EMULATION

Chedly Ben Azizi, Claire Guilloteau, Gilles Roussel, and Matthieu Puigt

Univ. Littoral Côte d'Opale, LISIC – UR 4491, F-62219 Longuenesse, France
{chedly.ben-azizi,claire.guilloteau,gilles roussel,matthieu.puigt}@univ-littoral.fr

## ABSTRACT

Generating high-fidelity hyperspectral data is an essential component of observation missions but traditionally involves expensive numerical simulations. This challenge can be addressed by constructing a statistical model approximating the distribution of simulated data. This process is referred to as emulation. This paper introduces a novel hyperspectral emulation approach pairing a Variational Autoencoder and a Neural Interpolator for fast hyperspectral image (HSI) emulation. The VAE is trained to reproduce data, and an interpolator is then used to link biophysical parameters to the latent space learned by the VAE. This method enables the sampling of HSI from biophysical parameters efficiently and accurately. This approach is evaluated on Sentinel-3 Ocean Colour Images and shows significant improvements in accuracy, efficiency, and scene-level information synthesis compared to existing methods.

**Index Terms**— hyperspectral imaging, emulation, remote sensing, variational autoencoder, deep learning

## 1. INTRODUCTION

Hyperspectral imaging captures a 2-D spatial and 1-D spectral description of a scene, providing a high spectral resolution data-cube, referred to as a hyperspectral image (HSI). Its applications extend across a diverse array of fields, e.g. astrophysics [1], agricultural monitoring [2], or remote sensing [3]. Generating high-fidelity data is crucial to the success of hyperspectral observation missions, whether for the early development of dedicated data processing algorithms or for the design of future missions. In the particular case of remote sensing, simulators capable of synthesising satellite images from biophysical parameters have been developed [4]. However, such modeling of complex non-linear physics generally requires expensive numerical simulations.

The recent and rapid development of machine learning techniques led to the creation of statistical models able to approximate such simulators with a high degree of accuracy for a fraction of the computational cost [5]. These models, called emulators, are in practice built using a set of input-output pairs. These can either be generated through a limited number of simulator runs, or taken from an observation mission, e.g. Sentinel-3 [3] for which no simulator is available. This training set is then used to learn the parameters of a probability model. Previous works on emulation focused on Bayesian statistical methods, mainly Gaussian processes [6], although other regression methods were later used. In the context of hyperspectral imaging, the use of machine learning regression algorithms (MLRAs) as powerful emulation methods was investigated in [7, 8, 9] in order to solve different plant spectroscopy tasks. Among these MLRAs, artificial neural networks (ANN) stand out thanks to their capacity to model complex nonlinear problems.

ANN—and deep learning in general—in hyperspectral imaging has been mainly used for classification tasks [10]. More recently, however, deep generative networks (DGN) were introduced to solve tasks such as spectral unmixing [11] or data fusion [12]. DGNs are models that represent the probability distribution of the training data, without necessarily evaluating the distribution function explicitly. Therefore, DGNs allow for the synthesis of realistic data by sampling from this distribution, which makes them adequate for emulation. One example is in [13], where a generative adversarial network (GAN) [14] was used to emulate high-fidelity weak lensing convergence maps. Another significant type of DGNs is the variational autoencoder (VAE) [15]. It uses learned approximate inference to model the data distribution and is trained by maximising the variational lower bound. Compared to GANs, VAEs are easier to train and yield an explicit (however approximate) density estimation. They are also efficient when dealing with high dimensional data, such as HSI. An example of their use for emulation in another context can be found in [16].

While existing HSI emulation methods have made significant contributions, they primarily focus on spectrum-to-spectrum synthesis, limiting their ability to capture the inherent complexities of scene-level information within HSI data.

THIS WORK IS PARTIALLY FUNDED BY THE ULCO RESEARCH POLE "MUTATIONS TECHNOLOGIQUES ET ENVIRONNEMENTALES" AND BY MAIA PIA 4. THIS WORK IS SUPPORTED BY THE GRADUATE SCHOOL IFSEA THAT BENEFITS FROM A FRANCE 2030 GRANT (ANR-21-EXES-0011) OPERATED BY THE FRENCH NATIONAL RESEARCH AGENCY.
Furthermore, to our knowledge, DGN-based HSI emulators are yet to be developed. In this work, we investigate the VAE capacity to emulate HSI, when applied to the data from Sentinel-3 Ocean Colour, by first training it to reproduce input images from a learned low dimensional latent representation, known as latent variables. Then, we train an interpolator based on a Convolutional Neural Network (CNN) [17] to map the bio-physical parameters to the learned latent space. Finally, emulating HSI from bio-physical parameters is presented here as mapping the latter to latent variables and decoding them to images. The two step process allows us to separate between synthesis and interpolation. Therefore, each component can be trained and validated independently, facilitating hyperparameter tuning. Additionally, we present a comprehensive evaluation of the emulated synthetic data, highlighting the significant improvements over existing methods in terms of accuracy, efficiency, and scene-level information synthesis.

The structure of the paper is organized as follows: Section 2 presents the proposed approach detailing the two-step emulation model. Section 3 outlines the experimental setup, and discusses the results obtained. Lastly, Section 4 offers a summary of the study's main findings and future research directions.

## 2. PROPOSED APPROACH

### 2.1. Dataset

Sentinel-3 is a series of Earth observation satellites developed and operated by the European Space Agency (ESA) and the European Organisation for the Exploitation of Meteorological Satellites (EUMETSAT). Its main mission is to retrieve high-quality environmental data and monitor Earth's health. To achieve this goal, Sentinel-3 satellites are equipped with multiple remote sensing instruments, namely the Ocean and Land Colour Instrument (OLCI), a pushbroom imaging spectrometer that measures solar radiation in 21 bands over the visible/near-infrared (NIR) domain with a spatial resolution of 300 m. The Sentinel 3 OLCI Ocean Colour dataset consists of radiance HSI (Level 1 products) [3] and the corresponding atmospherically corrected reflectances and biophysical parameters related to ocean colour (Level 2 products) [3]. Among the latter, 6 independent variables were selected as input of the emulator, i.e., chlorophyll-a concentration (Chl), total suspended matter concentration (TSM), diffuse attenuation coefficient (Kd), photosynthetically active radiation (PAR), aerosol Angstrom exponent (Å), and integrated water vapour (IWV) [3].

The used dataset consists of OLCI products acquired between 19 April 2022 and 23 April 2022 over the Atlantic Ocean, the Mediterranean Sea, the North Sea, and the English Channel. Images were first cropped to 128 × 128. Then, we only kept those which align with the following criteria: at least 40% of the pixels are water and a maximum of 20% of

pixels of other classes (cloud, land, snow, ice). This resulted in 100 000 pairs of biophysical parameters and HSI of which $S \approx 25 000$ were randomly selected for training and $B \approx 5 000$ for validation.

### 2.2. VAE--interpolator coupled emulation

<figure><img src="image_2.png" /><figcaption>**Fig. 1:** Proposed emulator: First, a VAE is trained to reproduce input HSI. Then, the encoder is discarded and an interpolator is trained to map biophysical parameters to the learned latent space. Finally, emulation is achieved by projecting given input onto the latent space and generating a cube using the decoder.</figcaption></figure>

Traditionally, HSI emulation is achieved by implementing a dimensionality reduction, e.g Principal Component Analysis, on the emulated spectra and then using an MLRA to predict the corresponding coefficients [7, 8, 9]. This is done in order to solve MLRAs inadequacy for dealing with high dimensional data such as HSI. Our choice for coupling a VAE and an interpolator for emulation is inspired by the aforementioned approach without necessarily separating the dimensionality reduction from the emulation process. Furthermore, each component can be trained and validated on its own, facilitating hyperparameter tuning and allowing us to train the VAE independently of the choice of biophysical parameters. Therefore, the decoder block of the VAE is reusable and changing the biophysical parameters will only result in retraining the interpolator. Finally, the VAE offers a more structured latent space than other state of the art generative models—namely GANs—which would make interpolation easier.

The proposed emulation method consists of two parts, as shown in Fig. 1. First, we train a VAE to reproduce each input HSI. It encodes data into a latent space and decodes from this space to reconstruct the original data using convolutional neural networks for the encoder and the decoder. For the encoder, we use ResNet-18 model [17], a well known CNN architecture that constitutes a good baseline for our work. The decoder is composed of the same layers but in a reverse order.
der. We apply the method in [18] which consists of an up-sampling followed by a convolution. This is done to mitigate the checkerboard artifacts resulting from the transpose convolution layer generally used for decoders. The VAE is trained on 400 epochs, yielding a powerful HSI model $D_{\theta}$ that can be used for data sampling, and can therefore be validated on its own. We use the implementation from [19] for our experiments.

Secondly, after validating the replication of the HSI, we discard the encoder part of the VAE and train an interpolator to link the biophysical parameters to the latent space. Interpolation is the projection of the input variables on a the latent space learned in the first step. The interpolator is therefore a parametric model ($f_{\theta'}$) of the latent distribution $p(z|\mathbf{Y})$ where $\mathbf{Y}$ is the respective input biophysical variables, $z$ are the latent variables and $\theta$ the model parameters. HSI are emulated by sampling from this distribution, given an input variable $\mathbf{Y}^{(i)}$, and feeding the sampled latent variables to the decoder. The training objective of the network is then:

$$ \min_{\phi'} \mathcal{L}'(\theta, \phi', \mathbf{Y}^{(i)}) = \min_{\phi'} \|\mathbf{I}^{(i)} - D_{\theta}(f_{\phi'}(\mathbf{Y}^{(i)}))\|_{2}. \quad (1) $$

Alternatively, we can train the interpolator by minimising the projection error which consists of comparing the interpolation of the biophysical parameters $f_{\theta'}(\mathbf{Y}^{(i)})$ to the encoding $E_{\phi}(\mathbf{I}^{(i)})$ of the corresponding HSI $\mathbf{I}^{(i)}$. We opted for the first method for the sake of implementation simplicity and computational efficiency. Additionally, minimising reconstruction error yields better reconstruction results.

# 3. EXPERIMENTS AND RESULTS

## 3.1. Evaluation

**Compared methods:** We compare our proposed approach—dubbed specVAE—to the methods introduced in [7], namely the Kernel Ridge Regression (KRR) and the Artificial Neural Network (ANN), as they were found in [7] to provide the best performance. They both consist of applying Principal Component Analysis (PCA) to the target spectra. Then the regression model is trained to predict the corresponding Principal Components (PCs) from an input vector consisting of the previously mentioned biophysical parameters. First we set the number of components $M$ to 2 such that the model explains ~ 99.5% of the variance. We gradually increased $M$, achieving the best results with $M = 12$. We could not train these models on the same conditions as described in Section 2.1. Since these methods emulate spectra, the training set would consist of more than 4 billion pairs of biophysical parameters and spectra. Training an ANN on such a large dataset wouldn't necessarily yield better results, since individual spectra in a given patch are close to each other. As for KRRs, they are not suitable for large datasets since training them consists of determining a matrix the size of the dataset and that would be computationally prohibitive. Therefore, we

train the ANN and the KRR respectively on 12 800 000 and 1 000 input-output pairs of biophysical parameters and spectra, while in our method they consist of biophysical parameters and HSI. In order to generate a full hyperspectral cube, spectra first have to be emulated individually.

**Metrics:** To validate our method, four criteria have to be considered: global, spatial and spectral reconstruction quality, and execution time. Quantifying the latter is trivial and only consists of calculating CPU and GPU times. To measure the quality of the reconstruction, we use a variety of metrics. Firstly, we use the mean squared error (MSE), as it is part of the loss function that we are minimising and thus gives a global idea of the quality of the emulation. Then, we calculate the Structural Similarity Index Measure (SSIM) [20] and the Spectral Angle (SA) [21] to evaluate the spatial and spectral reconstruction, respectively. SSIM is a measure of the visual quality of an image based on the idea that it is determined by its structure or shape. On the other hand, the SA consists of determining the angle between two spectra, represented by vectors of dimension equal to the number of bands. Finally, since emulation can be seen as a regression task, we use the $R^2$, which is a statistic that determines the linear correlation between the predicted and ground truth spectra. Therefore, by considering all of the previously mentioned metrics, we gain a more accurate understanding of the global, spatial and spectral quality of emulation.

## 3.2. Results

<table><caption>**Table 1:** Validation Results and Processing Times (s: seconds).</caption><thead><tr><th></th><th>MSE</th><th>SSIM (avg)</th><th>SA (avg)</th><th>R<sup>2</sup></th><th>CPU (s)</th><th>GPU (s)</th></tr></thead><tbody><tr><td>ANN</td><td>50.0067</td><td>0.8464</td><td>0.0588</td><td>0.6796</td><td>3.3271</td><td>0.8798</td></tr><tr><td>KRR</td><td>100.2000</td><td>0.7758</td><td>0.1177</td><td>0.4756</td><td><strong>3.1305</strong></td><td>-</td></tr><tr><td>specVAE</td><td><strong>48.8302</strong></td><td><strong>0.8944</strong></td><td><strong>0.0469</strong></td><td><strong>0.8849</strong></td><td>14.2600</td><td><strong>0.3559</strong></td></tr></tbody></table>

The performance of the emulator was evaluated on a subset of the validation dataset consisting of 250 HSI, corresponding to 4,096,000 spectra. Results are shown in Table 1. Our method, dubbed specVAE, achieves the best MSE, SSIM, and average SA with the ANN coming close in second place. KRR lags behind, especially for MSE where it scores two times worse than the other methods. While specVAE outperforms compared methods in terms of image quality metrics, it requires significantly more computation time on the CPU compared to ANN and KRR. However, it shows a notable advantage in GPU computation time—i.e., 2.5 times faster than ANN—while the metric is not applicable to KRR. This suggests that specVAE is more suitable for GPU-accelerated environments, which are nowadays almost universally deployed in computing servers.

To further investigate the performance of our emulator, we plot general statistics (mean, standard deviation (SD)) of the emulated spectra on top the ground truth spectra (Figure 2). It shows a near-perfect overlap between the statistics of our method and of the ground truth. These similarities are
<figure><img src="image_3.png" /><figcaption>**Fig. 2:** General statistics (mean, standard deviation) for validation and corresponding emulated spectra using specVAE (top), ANN (middle), and KRR (bottom).</figcaption></figure>

confirmed by the high $R^2$ score achieved (0.8849). However, the mean and SD of the KRR method exhibit a clear mismatch, whereas the ANN method shows consistency with the ground truth in terms of the mean, albeit failing to account for the same level of variability. To emphasize this point, we compare emulated spectra against the original Sentinel-3 spectra (Fig. 3). The spectra are color-scaled by Chlorophyll-a concentration in order to show the correlation between the biophysical variable and the corresponding spectra. The radiance spectra obtained with our specVAE emulator from Chlorophyll-a concentrations are better synthesised than the ANN and KRR emulators, especially from high Chl concentrations while they tend to be under-represented in the training dataset.

Finally, we visually inspect the spatial quality of the reconstruction by recreating RGB images from the HSI cubes, by using the CIE tristimilus functions [22]. Results are shown in Fig. 4. For ANN and KRR, ice pixels were masked since water related biophysical variables aren't relevant. Regardless, specVAE is not only able to recreate these pixels but also to discriminate between ice, cloud and land (examples not shown here). Furthermore, the colour of the water of the specVAE HSI sample, intrinsically related to the shape of the spectra, is much closer to that of the ground truth than the ANN and KRR HSI samples. However, as generally seen with VAEs and likelihood-based methods, the specVAE HSI are faintly blurred.

## 4. CONCLUSION

In this study, we have introduced a novel and efficient VAE-Interpolator coupled approach for hyperspectral image emulation, demonstrating substantial enhancements in accuracy,

<figure><img src="image_4.png" /><figcaption>**Fig. 3:** Original Sentinel-3 (a) and emulated spectra using specVAE (b), ANN (c), and KRR (d). The spectra are color-scaled by Chlorophyll-a concentration.</figcaption></figure>

efficiency, and scene-level information synthesis when compared to existing methods. Our method effectively projects biophysical parameters into a low-dimensional latent space and utilizes these projections to generate high-fidelity HSI. Moving forward, we propose to evaluate our method based on a practical setting such as biophysical parameter retrieval. Moreover, we intend to investigate several strategies to mitigate the blurring due to the use a standard VAE architecture for our emulator. Looking further, alternative Deep Generative Networks (DGN) are to be explored, such as Generative Adversarial Networks (GAN) and Ladder Variable Autoencoders (LVAE), as potential models for HSI emulation, with a focus on evaluating their accuracy and efficiency.

## 5. REFERENCES

[1] C. Guilloteau, T. Oberlin, O. Berné, and N. Dobigeon, "Hyperspectral and multispectral image fusion under spectrally varying spatial blurs-application to high dimensional infrared astronomical imaging," *IEEE Trans. Comput. Imaging*, vol. 6, pp. 1362-1374, 2020.

[2] A. A. Gowen, C. P. O'Donnell, P. J. Cullen, G. Downey, and J. M. Frias, "Hyperspectral imaging-an emerging process analytical tool for food quality and safety control," *Trends in food science & technology*, vol. 18, no. 12, pp. 590-598, 2007.

[3] C. Donlon, B. Berruti, A. Buongiorno, M.-H. Ferreira, P. Féménias, J. Frerick, P. Goryl, U. Klein, H. Laur, C. Mavrocordatos, et al., "The global monitoring for environment and security (gmes) sentinel-3 mission," *Remote sensing of Environment*, vol. 120, pp. 37-57, 2012.

[4] A. Börner, L. Wiest, P. Keller, R. Reulke, R. Richter, M. Schaepman, and D. Schläpfer, "Sensor: a tool for the
<figure><img src="image_5.png" /><figcaption>(a) Ground truth</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) specVAE
SSIM: 0.9118</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>(c) ANN
SSIM: 0.8155</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>(d) KRR
SSIM: 0.7562</figcaption></figure>

<caption>**Fig. 4:** RGB compositions of original Sentinel-3 (a) and emulated scenes using specVAE (b), ANN (c), and KRR (d). For ANN and KRR, ice pixels (white) are masked.</caption>

simulation of hyperspectral remote sensing systems,” *Photogramm. Eng. Remote Sens.*, vol. 55, no. 5-6, pp. 299–312, 2001.

[5] M. C. Kennedy and A. O’Hagan, “Predicting the output from a complex computer code when fast approximations are available,” *Biometrika*, vol. 87, no. 1, pp. 1–13, 2000.

[6] J. L. Gómez-Dans, P. E. Lewis, and M. Disney, “Efficient emulation of radiative transfer codes using gaussian processes and application to land surface parameter inferences,” *Remote Sensing*, vol. 8, no. 2, pp. 119, 2016.

[7] J. Verrelst, J. P. Rivera Caicedo, J. Muñoz-Marí, G. Camps-Valls, and J. Moreno, “Scope-based emulators for fast generation of synthetic canopy reflectance and sun-induced fluorescence spectra,” *Remote Sensing*, vol. 9, no. 9, pp. 927, 2017.

[8] J. Verrelst, J. P. Rivera Caicedo, J. Vicent, P. Morcillo Pallarés, and J. Moreno, “Approximating empirical surface reflectance data through emulation: opportunities for synthetic scene generation,” *Remote sensing*, vol. 11, no. 2, pp. 157, 2019.

[9] M. Morata, B. Siegmann, P. Morcillo-Pallarés, J. P. Rivera-Caicedo, and J. Verrelst, “Emulation of sun-induced fluorescence from radiance data recorded by the hyplant airborne imaging spectrometer,” *Remote sensing*, vol. 13, no. 21, pp. 4368, 2021.

[10] S. Li, W. Song, L. Fang, Y. Chen, P. Ghamisi, and J. A. Benediktsson, “Deep learning for hyperspectral image classification: An overview,” *IEEE Trans. Geosci. Remote Sens.*, vol. 57, no. 9, pp. 6690–6709, 2019.

[11] F. Hadi, J. Yang, M. Ullah, I. Ahmad, G. Farooque, and L. Xiao, “DHCAE: deep hybrid convolutional autoencoder approach for robust supervised hyperspectral unmixing,” *Remote Sensing*, vol. 14, no. 18, pp. 4433, 2022.

[12] J. Ma, W. Yu, C. Chen, P. Liang, X. Guo, and J. Jiang, “Pan-GAN: An unsupervised pan-sharpening method for remote sensing image fusion,” *Information Fusion*, vol. 62, pp. 110–120, 2020.

[13] M. Mustafa, D. Bard, W. Bhimji, Z. Lukić, R. Al-Rfou, and J. M. Kratochvil, “CosmoGAN: creating high-fidelity weak lensing convergence maps using generative adversarial networks,” *Computational Astrophysics and Cosmology*, vol. 6, no. 1, pp. 1–13, 2019.

[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” *Communications of the ACM*, vol. 63, no. 11, pp. 139–144, 2020.

[15] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” arXiv e-print arXiv:1312.6114, 2022.

[16] C. Guilloteau, N. Ramachandra, F. Lanusse, S. Hassan, Y.-s. Ting, M. Huertas-Company, et al., “Generative networks for emulating synthetic sky images,” *Technical report, Kavli Summer Program in Astrophysics*, 2019.

[17] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in *Proc. IEEE CVPR*, 2016, pp. 770–778.

[18] A. Odena, V. Dumoulin, and C. Olah, “Deconvolution and checkerboard artifacts,” *Distill*, 2016.

[19] A. Subramanian, “Pytorch-vae,” https://github.com/AntixK/PyTorch-VAE, 2020.

[20] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” *IEEE Trans. Image Process.*, vol. 13, no. 4, pp. 600–612, 2004.

[21] R. H. Yuhas, A. F. Goetz, and J. W. Boardman, “Discrimination among semi-arid landscape endmembers using the spectral angle mapper (SAM) algorithm,” in *JPL, Summaries of the Third Annual JPL Airborne Geoscience Workshop. Volume 1: AVIRIS Workshop*, 1992.

[22] T. Smith and J. Guild, “The cie colorimetric standards and their use,” *Transactions of the optical society*, vol. 33, no. 3, pp. 73, 1931.