# Differentiable chief-ray tracing simulator for coded-aperture spectral imaging

Léo Paillet, Antoine Rouxel, Hervé Carfantan, Antoine Monmayrant, Simon Lacroix

► To cite this version:

Léo Paillet, Antoine Rouxel, Hervé Carfantan, Antoine Monmayrant, Simon Lacroix. Differentiable chief-ray tracing simulator for coded-aperture spectral imaging. Computational Optical Sensing and Imaging, Optica Publishing Group, Jul 2024, Toulouse, France. pp.CTh4B.2, 10.1364/COSI.2024.CTh4B.2. [https://hal.science/hal-04692877v1](https://hal.science/hal-04692877v1)

HAL Id: hal-04692877

Submitted on 24 Sep 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Differentiable chief-ray tracing simulator for coded-aperture spectral imaging

L. Paillet<fnref n="1" />,<fnref n="2" />*, A. Rouxel<fnref n="1" />, H. Carfantan<fnref n="2" />, A. Monmayrant<fnref n="1" />, and S. Lacroix<fnref n="1" />

**Abstract:** We introduce an end-to-end differentiable simulation framework for designing Coded-Aperture Snapshot Spectral Imagers (CASSI) and exploring acquisition strategies. By leveraging automatic differentiation, we dimension the optical system and optimize the coded aperture, enhancing imaging quality and facilitating effective co-design of the instrument. © 2024 The Author(s)

## 1. Introduction

While computational imaging is making significant strides, it still encounters challenges stemming from the sequential design paradigm and accessibility of open-source tools. Unlike conventional imaging systems, computational imaging systems present a high level of intricacy between optical hardware and processing software, thus highlighting the drawbacks of independently developing these components. Despite existing literature proposing end-to-end solutions for CASSI systems [1], these often rely on simplistic sensing matrix models. The advent of differentiable programming bridges the gap between physics-based modeling and deep learning, facilitating effective gradient flow and global optimization without compromising optical fidelity. However, new differentiable tools like *dO* [2] fall short in efficiently evaluating high-resolution hyperspectral coded images as they rely on the time-consuming ray-surface intersection method.

We introduce a novel fully differentiable simulation framework (see figure 1-A) that leverages a chief-ray tracing optical model [3] for the design of CASSI systems with a new compromise between speed and accuracy. While our primary focus is here scene classification or reconstruction from coded images, our framework allows for an end-to-end optimization of CASSI systems for a wide variety of imaging tasks. To illustrate the capability of this framework, we detail the co-design of a double-Amici-prism dual-disperser CASSI.

## 2. Optical design with gradient-descent

Our framework [4] employs a chief-ray tracing optical model for the evaluation of the sensing matrix $H_Φ$, complemented by nearest neighbor interpolation for the generation of coded images on the detector plane. The model's analytical nature ensures rapid chief-ray propagation while incorporating geometric distortions from the dispersive element into the sensing matrix evaluation. The entire process has been implemented in PyTorch to take full advantage of the library's batch processing and automatic differentiation capabilities, ensuring efficient simulations.

Gradient-descent optimization targets both the defining properties of the spectral disperser<fnref n="1" /> and its precise alignment within the optical setup. This optimization is governed by a cost function $L_{optic}$ that accounts for spectral dispersion, system compactness, constraints direct-view, and minimizes geometric distortions. Our methodology achieves a spectral dispersion in the coded aperture plane of 2175 µm for the [450 – 650] nm range (thus around 1 % deviation from the desired dispersion of 2200 µm) with the prism design described in figure 1-B and a 75-mm focal length lens. As shown in figure 1-B and contrarily to widely used simplified model [1], residual geometric distortions in the coded aperture plane are properly accounted for in the sensing matrix evaluation.

## 3. Joint-optimization of the coded-aperture and algorithm

In this study, we detail for a dual-disperser CASSI the co-design process of both the coded-aperture pattern $Φ$ and the parameters $θ$ of the cutting-edge transformer-based reconstruction algorithm DAUHST [5].

Initially, we decouple the optimization to establish a baseline, validating effective hyperspectral scene reconstruction from coded measurements using randomly generated binary patterns. This is accomplished by training the reconstruction algorithm $M_θ$ on the CAVE 1024 dataset [6]. Subsequently, we engage in a simultaneous optimization process involving both the coded aperture and the algorithm parameters. This is framed as an optimization

<footnote n="1">¹ LAAS-CNRS, Université de Toulouse, CNRS, 7 avenue Colonel Roche, 31400, Toulouse</footnote>

<footnote n="2">² IRAP, Université de Toulouse, UPS/CNRS/CNES, 14 Avenue Édouard Belin, 31400, Toulouse</footnote>

<footnote n="3">³lpaillet@laas.fr</footnote>

<footnote n="1">¹In the present case, both the glass materials and the apex angles of a double-Amici prism.</footnote>
<figure><img src="image_2.png" /><figcaption>Fig. 1: A: end-to-end framework for coded-aperture spectral imaging systems including design constraints at each step of the simulation through multiple loss functions; B: Double-Amici prism geometric distortions for a 2200 µm spectral dispersion: C: Reconstruction results for the CAVE 1024 dataset.</figcaption></figure>

problem aimed at minimizing the reconstruction error across a set of K scenes $S_k$, defined as follows:

$$ \{\Phi^*, \theta^*\} \in \arg\min_{\Phi, \theta} \mathcal{L}_{\text{proc}}(\Phi, \theta|S, d) \quad \text{with} \quad \mathcal{L}_{\text{proc}}(\Phi, \theta|S) = \frac{1}{K} \sum_{k=1}^{K} \text{RMSE}(\mathcal{M}_{\theta}(H_{\Phi}S_k), S_k) \quad (1) $$

We present optimization results for both binary and non-binary coded aperture designs as they are two current use-cases in coded-aperture spectral imaging [1]. We utilize a step-through estimator to achieve aperture binarization, facilitating uninterrupted gradient flow during optimization. Comparative analysis, as outlined in the table of figure 1-C, demonstrates superior performance metrics for optimized aperture designs $\Phi$, underscoring our framework's efficacy as a comprehensive tool for end-to-end design in spectral imaging.

## 4. Conclusion

In this work, we harness the power of automatic differentiation to construct a holistic framework for the end-to-end design of coded-aperture spectral imaging systems, ensuring the retention of high optical fidelity throughout the process. Our approach enables the dimensioning of the imaging system's dispersive element, adhering to both optical and system constraints, and facilitates the concurrent optimization of the coded aperture patterns and the parameters of a state-of-the-art reconstruction algorithm.

Future work includes gathering an appropriate dataset for performing a global optimization, including dispersive element parameters, on task-specific metrics. Additionally, we are in the process of refining our optical model's precision by incorporating a shift-variant point-spread function [7]. This improvement is being achieved through the adoption of an implicit neural representation, striking a balance between inference speed and image quality.

## References

1. Jorge Bacca, et al. Deep Coded Aperture Design: An End-to-End Approach for Computational Imaging Tasks. *IEEE Transactions on Computational Imaging*, 7:1148–1160, 2021.

2. Congli Wang, et al. dO: A Differentiable Engine for Deep Lens Design of Computational Imaging Systems. *IEEE Transactions on Computational Imaging*, 8:905–916, 2022.

3. Antoine Rouxel, et al. Accurate ray-tracing optical model for coded aperture spectral snapshot imagers. *Applied Optics*, 63(7):1828, February 2024.

4. Antoine Rouxel et al. SIMCA. https://github.com/a-rouxel/simca, 2023.

5. Yuanhao Cai, et al. Degradation-Aware Unfolding Half-Shuffle Transformer for Spectral Compressive Imaging, 2022.

6. Fumihito Yasuma, et al. CAVE Dataset. https://www.cs.columbia.edu/CAVE/databases/multispectral/, 2008.

7. Ethan Tseng, et al. Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. *ACM Transactions on Graphics*, 40(2):1–19, April 2021.