# The Marginal Importance of Distortions and Alignment in CASSI systems

Léo Paillet, Antoine Rouxel, Hervé Carfantan, Simon Lacroix and Antoine Monmayrant

***Abstract***—Coded Aperture Snapshot Spectral Imagers (CASSI) are designed to acquire 2D coded acquisitions of hyperspectral scenes (HSS). Associated with reconstruction algorithms, they allow to analyze the HSS using a small amount of data. Various configurations of such imagers exist, with different distortions. In this article we show that reconstruction quality is relatively insensitive to these distortions, if modeled correctly. To this end, we introduce a differentiable ray-tracing-based model that incorporates aberrations and distortions to render realistic coded hyperspectral acquisitions using CASSI. Such simulated acquisitions are used to train state-of-the-art hyperspectral cube reconstruction algorithms. We also adapted these algorithms through the use of a ray-tracing-based mapping function which accounts for aberrations and distortions. We evaluated four comparable CASSI systems with varying degree of optical aberrations and misalignments, using five state-of-the-art hyperspectral cube reconstruction algorithms. Our analyses show that if properly modeled, the effects of geometric distortions of the system and misalignment of the dispersive elements have a marginal impact on the overall reconstruction quality. Therefore, relaxing traditional constraints on measurement conformity and fidelity to the scene enables the development of novel imaging instruments, guided by performance metrics applied to the design or the processing of acquisitions. By providing a complete framework for design, simulation and evaluation, this work contributes to the optimization and exploration of new CASSI systems and more generally to the computational imaging community.

***Index Terms***—Compressive Sensing, Coded-Aperture, Hyperspectral Imaging, CASSI, Optical Design, Ray-tracing

## I. INTRODUCTION

TRADITIONAL hyperspectral imagers render three-dimensional data cubes by scanning the scenes along a spectral or spatial dimension [1]. This leads to two main drawbacks: the transmission and processing of a large quantity of data, and a limitation to static scenes. Snapshot hyperspectral coded aperture imaging systems [2]–[6] rely on compressed sensing theory [7] to address these shortcomings. Specifically, CASSI systems [3], [6] reduce the redundancy present in hyperspectral scenes (HSSs) by performing encoding of the contained information. Out of the many different implementations of CASSI, the two most common are DD-CASSI [6] and SD-CASSI [3], [8]. Thanks to the double disperser, DD-CASSI induces mainly spectral encoding, while SD-CASSI induces both spectral and spatial encoding. These systems use a coded aperture (or “mask”) and dispersive elements to

spectrally and spatially modulate the HSS. The coded aperture can be optimized for specific applications [9], but can also be used as a random sampler [8], [10].

A primary goal of coded hyperspectral imaging is the reconstruction of the full hyperspectral cube (HSC), enabling further processing of the three-dimensional data cube. Historically, this reconstruction from coded acquisitions predominantly relied on model-based methods [10]–[14]. While these methods offer insight into the reconstruction process, they suffer from long reconstruction times and suboptimal reconstruction quality. Recently, deep learning approaches have provided fast reconstruction times and high-quality results, albeit with less interpretability [9], [15]–[17]. Transformers [18] have also achieved superior results by leveraging non-local spatial relationships between pixels [19], [20], resulting in better exploitation of spatial and spectral interrelationships in HSSs.

However, state-of-the-art reconstruction algorithms [19], [21]–[24], while primarily data-driven, also integrate a simplified representation of the propagation model. They assume that CASSI systems produce spatially uniform dispersion, typically linear, without any optical misalignment. This approximation is best suited for grating-based CASSI [25]–[28], but most CASSI systems are using prism-based dispersive element that offer higher efficiency for large spectral ranges. This approximation is also well suited for most CASSI systems that employ a double-Amici prism as the dispersive element [8], [29]–[33]. Compared to single prisms, the advantage of double-Amici prism assemblies is twofold: they allow for a direct-view geometry and do not exhibit anamorphic or optical distortions. Still, double-Amici prisms require longer manufacturing times, are more expensive, and misalignments must still be addressed when working with prototypes. Alternatively, using a single prism as the dispersive element results in a peculiar arrangement of information in the captured images, which can be properly exploited for reconstruction if the optical model is precise and accurate.

In this article, we evaluate the impact of distortions and misalignments on the coded information, when the optical model is properly considered. For this matter, we need a CASSI simulator accounting for both distortions and the spatially and spectrally Point Spread Function (PSF). Some image formation models [29] include PSF but they depend on calibration data, necessitating the assembly and calibration of a prototype. Recently, we proposed an accurate chief-ray-based propagation model [34] for dimensioning and optical distortions estimation, but it does not account for optical aberrations and cannot be used for Monte-Carlo rendering or PSF estimation. Recent works in computational imaging

<footnote>This work was supported by the CNRS and the Université de Toulouse. Léo Paillet is with LAAS-CNRS, Toulouse, France and with IRAP, Toulouse, France (e-mail: leo.paillet@laas.fr). Antoine Rouxel, Simon Lacroix and Antoine Monmayrant are with LAAS-CNRS, Toulouse, CNRS, France (e-mails: firstname.name@laas.fr). Hervé Carfantan is with IRAP, Toulouse, France (e-mail:Herve.Carfantan@irap.omp.eu).</footnote>
utilize differentiable ray-tracing to enable end-to-end optical design [35]–[37], in particular DiffOptics (dO) [37]. However, to the best of our knowledge, these works only address optical systems with axial symmetry thus cannot be used for coded aperture hyperspectral systems.

We propose an extension of dO to more general optical systems – in particular breaking axial symmetry – which thus enables accurate modeling of optical behavior and precise rendering of coded hyperspectral acquisitions.

Although it does not simulate the PSFs, we can harness the ray-traced spot diagrams to render accurate coded acquisitions encompassing distortions and aberrations. By leveraging ray tracing, we create an accurate spectral and spatial mapping between the object and image planes of SD-CASSI systems. This mapping is then introduced in the reconstruction algorithms. This approach facilitates seamless processing across different CASSI systems, regardless of distortions or misalignments. We evaluate four distinct SD-CASSI configurations, constructed with either a double-Amici prism assembly or a single prism. To assess the impact of misalignments, two of these configurations are deliberately misaligned. The reconstructions from these four configurations are compared using five reconstruction algorithms and standard evaluation metrics.

Our results demonstrate that distortions and misalignments have a marginal impact on the information encoded in SD-CASSI acquisitions, and hence on the reconstruction quality, provided the rendering is realistic and an accurate model is incorporated into the reconstruction. This finding implies that the choice and fine-tuning of the reconstruction algorithm are more critical than the specific optical system.

The contributions of our work are:

* We implement coded aperture hyperspectral optical systems within a differentiable ray-tracing framework, enabling the rendering of distorted and aberrated images from backward ray-tracing and facilitating the end-to-end design of such systems before prototyping.

* We devise an accurate spatio-spectral mapping based on a realistic propagation model and introduce it, as a spatio-spectral prior, in the state-of-the-art reconstruction algorithms. This approach leverages two-dimensional measurements from the system and the associated rendering model.

* We highlight the marginal importance of distortions and misalignments in CASSI systems for acquisition processing purposes, provided the reconstruction process incorporates an accurate propagation model.

The paper organization is described as follows. Section II starts by explaining the design of a double-Amici prism assembly with the same angular spectral spreading than an off-the-shelf single prism, but none of its optical distortions. Two comparable SD-CASSI systems are then simulated – one based on the single prism, one on the double-Amici prism assembly–, and two configurations are considered for each: one with a perfectly aligned prism and one with a purposefully misaligned prism. These four configurations allow for the validation of our differentiable ray-tracer by comparing the obtained spot diagrams with a well-known ray-tracer (Zemax). The section ends with the rendering process

<figure><img src="image_1.png" /><figcaption>Fig. 1: Top-view of the systems layout for the (SP) configuration (a) and the (AP) configuration (b). An example of a coded aperture and a zoom on the detector planes (c). The imaging lens is not pictured here, we consider that the scene is perfectly imaged on the coded aperture's plane.</figcaption></figure>

of realistic CASSI acquisitions for each configuration through our ray tracer. Afterwards, Section III details the processing of coded acquisitions to reconstruct HSCs with algorithms taking into account the propagation model. Reconstruction quality is finally compared across all four configurations, for five state-of-the-art algorithms, demonstrating that the performance metrics are similar across configurations for a given algorithm.

All the source code about our modified version of dO, the adapted networks, and how to generate the figures is available at https://github.com/lpaillet-laas/DiffCassiSim.

## II. OPTICAL DESIGN AND RENDERING

Given that CASSI systems performances cannot be fairly assessed and compared using simple propagation models [10], [38], [39], which overlook the geometric distortions caused by dispersive elements as well as misalignment, we have developed a differentiable ray-tracing-based renderer based on dO, that takes into account distortions, misalignments and aberrations.

We applied it to generate accurate rendering of the coded acquisitions of two single disperser (SD) CASSI systems: one system using a single prism as the dispersive element, referred as (SP), and one with a double-Amici prism assembly, referred as (AP), shown in Figure 1. The two other configurations are duplicates for which the dispersive element has been purposefully misaligned by 5° around the x-axis with respect to the reference frames of Figure 1, and respectively referred as (mSP) and (mAP). We chose an exaggerated misalignment, far larger than what an experimenter should achieve, in order to consider the worst-case scenario and prove that even a large misalignment can lead to satisfactory processing and hence be considered marginal.

In this section, we delve into:

* The design process of a custom double-Amici prism to minimize optical distortions and achieve direct-view.
* The implementation of the four optical configurations within dO, highlighting how different configurations are simulated.

* The methodology used for generating realistic coded hyperspectral acquisitions.

## A. Case Study: SD-CASSI

The (SP) system illustrated in Figure 1-(a) contains a first lens to collimate the light onto the prism and a second one to image the light onto the detector. For the (AP) system, as for most recent Amici-based systems [8], [30], a single relay lens is used in a $2f - 2f$ configuration, with the dispersive element positioned between the lens and the detector, as shown in Figure 1-(b).

To ensure a fair comparison between both systems, it is crucial to dimension the dispersive elements so that they exhibit comparable spectral dispersion. Our analysis is conducted on systems with a detector comprising 512 × 512 pixels with a 10-µm pitch resulting in a ≃ 5 × 5 mm² field of view, a spectral range of [450 - 650] nm centered at 520 nm, and a numerical aperture of 0.05. The systems utilize lenses with a focal length of $f = 50$ mm. The targeted angular spectral spreading for these systems is $\Delta_0 = 0.95^\circ$, resulting in a spatio-spectral spreading $S$ on the detector of $S \approx 830$ µm (83 pixels) for the central point of the field of view. These parameters are selected based on the following criteria: the expected number of resolved pixels across the field of view, the consistency with the spectral range and resolution of the KAIST dataset [40], and the compatibility with standard off-the-shelf components.

## B. Design of a Double-Amici Prism for Fair Comparison

Our first goal is to design a double-Amici prism that replicates the angular spectral dispersion $\Delta_0 = 0.95^\circ$ of a standard N-BK7 equilateral prism at minimum deviation $D_0$ illustrated in Figure 2-(a), while minimizing geometric distortions and ensuring a direct-view configuration.

To design the prism, we employ the chief-ray-based dimensioning tool SIMCA, described in [41], [42], which solves a gradient-based optimization problem incorporating both optical and system-related parameters. Although dO is more powerful for differentiable optical design, we chose SIMCA as it was specifically developed for CASSI systems, thus offering more straightforward and faster optimization.

1) **Prism Parametrization:** We optimize the following five parameters, illustrated in Figure 2-(b): $\alpha_c$ the angle of incidence on the double-Amici prism assembly, $A_1$ and $A_2$ the apex angles of the first and second prisms, respectively, and $m_1$ and $m_2$ the glass materials for the two prisms, selected from the Schott catalog. Since the glass materials are discrete and not differentiable by default, we model the dispersion curve of each glass using two continuous sub-parameters: the refractive index at the "d" Fraunhofer line and the Abbe number, following the methodology described in [35], [43]. During optimization, we treat these sub-parameters as continuous variables. After optimization, we select the glass materials from the catalog that are closest to the optimized sub-parameter values.

<figure><img src="image_2.png" /><figcaption>Fig. 2: Optical designs of dispersive elements in our imaging system. (a) A commercially available N-BK7 equilateral prism, aligned in for minimum deviation $D_0$. (b) A custom-designed double-Amici prism optimized to minimize distortions and achieve direct-view geometry. Both designs exhibit comparable angular spectral spreading $\Delta$ to ensure a fair comparison of system performance.</figcaption></figure>

2) **Loss Functions:** To optimize the prism design, we first aim to match the angular spatio-spectral dispersion of the base design $\Delta_0$. We then minimize the optical distortions across the field of view and for the whole spectral range. We finally enforce an easy-to-align and compact system by minimizing the angular deviation $D$ and the thickness of the prism.

The spectral dispersion loss $\mathcal{L}_\Delta$ is defined as the squared difference between the base spectral dispersion $\Delta_0$ and the dispersion $\Delta$ calculated for the current prism design. We compute $\Delta$ as the absolute difference between the output angles after the prism for the shortest and longest wavelengths at the center of the field-of-view, respectively denoted as $\Delta_{\lambda_{min}}$ and $\Delta_{\lambda_{max}}$:

$$ \mathcal{L}_{\Delta} = (\Delta_0 - |\Delta_{\lambda_{max}} - \Delta_{\lambda_{min}}|)^2 \quad (1) $$

The distortion loss $\mathcal{L}_\varepsilon$ quantifies the geometric distortions introduced by the prisms. We calculate a distance tensor $\varepsilon$ which measures how each imaged point of the scene is displaced due to distortion. In practice, we measure the Euclidean distance between the distorted coordinates $(X_{ds}, Y_{ds})$ and the ideal coordinates $(X_{id}, Y_{id})$:

$$ \varepsilon_{i,j,k} = \sqrt{(X_{ds}^{i,j,k} - X_{id}^{i,j,k})^2 + (Y_{ds}^{i,j,k} - Y_{id}^{i,j,k})^2} \quad (2) $$

Here, $(X_{id}^{i,j,k}, Y_{id}^{i,j,k})$ are the coordinates of the ideal (undistorted) image grid points as described in [34]. We then define the distortion loss as the square of the maximum value in this distance tensor:

$$ \mathcal{L}_{\varepsilon} = \left( \max_{i,j,k} \varepsilon_{i,j,k} \right)^2 \quad (3) $$

The minimization of $\mathcal{L}_\varepsilon$ reduces the maximum geometric distortion across all points in the image grid.

The deviation loss $\mathcal{L}_D$ accounts for the total angular deviation induced by the prism configuration. The deviation is computed based on the chief-ray angles $\alpha_c$ and $\alpha_c^{\text{out}}$ (incidence angle and output angle of the chief-ray at the central wavelength) and the apex angles of the prisms. The deviation loss is then given by the squared total deviation:

$$ \mathcal{L}_D = (\alpha_c + \alpha_c^{\text{out}} + 2A_1 - A_2)^2 \quad (4) $$
where $A_i$ are the apex angles of the prisms.

The thickness loss $\mathcal{L}_t$ approximates the physical thickness of the double-Amici prism. It is proportional to the sum of the squared apex angles:

$$ \mathcal{L}_t = 2A_1^2 + A_2^2 \qquad (5) $$

This loss helps to minimize the overall size and weight of the optical system.

The glass distance loss ($\mathcal{L}_g$) measures the squared difference between the refractive index ($n_d$) and Abbe number ($v_d$) of the selected glass materials and the closest available materials in the Schott catalog. This loss ensures that the materials chosen during the optimization process are realistic and match closely with available catalog materials. For a double-prism system, the glass distance loss is defined as:

$$ \mathcal{L}_g = \gamma_g \cdot \min_{\text{glass}} \left( \left( \frac{n_d - n_d^{\text{catalog}}}{\Delta n_d} \right)^2 + \left( \frac{v_d - v_d^{\text{catalog}}}{\Delta v_d} \right)^2 \right) \qquad (6) $$

where $\Delta n_d$ and $\Delta v_d$ represent the ranges of refractive index and Abbe number in the Schott catalog. $\gamma_g$ is a weight parameter, currently set to $10^4$, enabling comparable values between losses inside the global composite loss. This ensures that the chosen materials are practical and manufacturable.

Finally, the total internal reflection loss ($\mathcal{L}_R$) penalizes designs that approach the critical angle for total internal reflection. Although this metric can be further refined, it is currently defined as:

$$ \mathcal{L}_R = \log (1 + \exp(a - D_{\text{TIR}}))^b \qquad (7) $$

where $D_{\text{TIR}}$ is the angular distance from which total internal reflection would occur and a and b are tunable parameters (currently set to 5 and 6, respectively).

3) Design Optimization: The goal is to find the set of design parameters $\theta = \{\alpha_c, A_1, A_2, m_1, m_2\}$ that minimize the total loss $\mathcal{L}(\theta)$ defined as a linear combination of the 6 loss terms:

$$ \mathcal{L} = w_{\Delta} \mathcal{L}_{\Delta} + w_{\varepsilon} \mathcal{L}_{\varepsilon} + w_{D} \mathcal{L}_{D} + w_{t} \mathcal{L}_{t} + w_{g} \mathcal{L}_{g} + w_{R} \mathcal{L}_{R} \qquad (8) $$

$\mathcal{L}$ encapsulates both optical and system-related objectives, allowing for a comprehensive optimization of the prism design. The weights $w$ in the loss expression (8) balance the contributions of each loss term according to their importance in the design objectives, they have been empirically chosen to guide the optimization. Specifically, the values used in the optimization were $(w_{\Delta}, w_{\varepsilon}, w_{D}, w_{t}, w_{g}, w_{R}) = (1, 1, 25, 5, \text{iteration step}, 10)$, with all distances in mm and angles in mrad. It is important to note that the choice of weights and loss functions should be tailored according to the specific design requirements. Although not utilized in this study, exhaustive grid searches or Monte Carlo-based methods could be necessary for determining optimal weights and loss functions tailored to more demanding design constraints.

4) Results: Using the Adam optimizer [44] and starting from the Amici parameters described in [8], we design a double-Amici prism assembly that meets our spectral dispersion requirement $\Delta_0$ of 0.95° at the detector plane within ±2%. The final materials for the Amici prism are N-SK2

and SK10 for $m_1$ and $m_2$, respectively. The apex angles are $A_1 = 29.2°$ and $A_2 = 47.9°$, with the incident chief-ray arriving at the prism at $\alpha_c = 5.1°$. Figure 3 presents the resulting spatial dispersion in the detector plane for the (AP) and the (SP). This spreading corresponds to the wavelength-dependent spatial shift of the image of the central point. As one can see, the curves have slightly different non-linearities, up to 7.9 pixel difference to the fitted linear spreading for (SP) and 11 pixel for (AP) (see Supplementary Figure 16). Yet the overall spreading ranges on the detector are similar: 830 µm for (SP) and 813 µm for (AP). Both systems thus exhibit comparable spectral dispersion.

<figure><img src="image_3.png" /><figcaption>Fig. 3: Comparison of the spatial dispersion at the detector (spreading) for (AP) (blue) and (SP) (orange). The dots correspond to 28 considered wavelengths over our spectral range.</figcaption></figure>

In terms of distortions however, both systems are extremely different. With respect to the (SP) setup, the maximum geometric distortion in (AP) is reduced from 214 µm to 6 µm and the mean distortion from 75 µm to 1.8 µm over a 5×5 mm field of view. Figure 4 shows the distortion map for both systems (SP) and (AP), and also for the misaligned configurations (mSP) and (mAP). The interpretation of these distortions in terms of keystone and smile is further discussed in section II-C and Figure 17 of the Supplementary.

This distortion minimization is achieved while maintaining a direct-view geometry with a total deviation $D$ of 0.1 mrad. The spatio-spectral spreadings on the detector equal 830 µm and 813 µm respectively for the (AP) and (SP) setups.

Both the commercially available N-BK7 equilateral prism and our custom-designed double-Amici prism assembly exhibit comparable spectral dispersion while presenting different geometric distortions, especially the substantial smile distortion induced by the equilateral prism, ensuring a fair comparison of systems performances in section III-D.

### C. Differentiable Simulation for SD-CASSI systems

Both single prism and double Amici prism assembly are employed to implement SD-CASSI systems using the differentiable optics design tool dO.

Figure 1 shows the two system's configurations in dO. As stated in section II-A, (AP) and (mAP) use a single relay lens (see Figure 1-(b)). Typically, an achromatic and field-corrected
<figure><img src="image_4.png" /><figcaption>Fig. 4: Distortion maps of a regular grid of points traced through the four considered optical systems, for the extrema and center wavelengths (450 nm, 520 nm, and 650 nm). Note the different colorscales between the configurations.</figcaption></figure>

relay lens is utilized to ensure high spatial resolution across the field of view and the entire spectral range. The compound relay lens is modeled as an ideal thin lens to isolate the impact of geometric distortions caused by the prism. This modeling choice avoids conflating prism-induced anamorphic distortions with lens-induced aberrations, ensuring that any observed deformation in the reconstruction arises from the system configuration. In the single-prism-based configurations (SP) and (mSP), two lenses are on either side of the prism (see section II-A). As proposed in [45], it is advantageous to minimize aberrations early in the setup by employing an achromatic objective lens before the prism and a simple doublet lens afterwards to reduce costs. Similar to the approach used for the Amici system, we model the achromatic objective lens as an ideal thin lens and implement a Thorlabs AC254-050-A-ML doublet lens for the second lens, as illustrated in Figure 1-(a).

The misaligned configurations (mSP) and (mAP) have been simulated by rotating the dispersive element by 5° around the x-axis as stated in Section II.

1) *Differentiable Ray-Traced Simulations:* dO supports both forward and backward ray-tracing, enabling the optimization of the design of imaging systems and the rendering of scenes through these systems. It was originally developed for designing free-form optics RGB imaging systems with axial symmetry: adapting it to CASSI systems required modifications due to the loss of axial symmetry caused by the prism. These modifications included the addition of a prism optical element, based on dO built-in surfaces, and the ability to

perform smooth rotations between optical elements, ensuring accurate modeling and simulation of the CASSI architecture.

In our modified dO, each optical element is then treated as an independent system, and rays are traced between these systems. The optical elements are composed of surfaces separated by materials, as in Zemax. Each element can then be rotated and shifted individually to closely match the design specifications.

The implementation of SD-CASSI designs in dO enables the acquisition of hyperspectral and coded hyperspectral images that realistically reflect the given optical system, in contrast to the simplified mathematical models used in [3], [11], [39]. Such models often struggle to account for all distortions and dispersions present in real optical systems. Our approach is more aligned with the mathematical models of [29], [46]. In addition, it produces spot diagrams through accurate ray tracing and can account for spectral dispersions along the *y* (vertical) axis and for the spectral dispersions continuity thanks to Monte-Carlo methods. This is particularly evident in the case of misaligned optical configurations, as seen in the (mAP) and (mSP) configurations (refer to Figure 4 and Supplementary Figure 17) although the Amici-based system is more tolerant to misalignment.

Additionally, dO supports the differentiable optimization of these designs, allowing fine-tuning of rotations, shifts between optical elements, and adjustments of surface distances and configurations. This functionality allows both the design and operational use of SD-CASSI systems within the same application, facilitating seamless imaging co-design.

2) *Validation of the Implementation in dO:* To validate our dO implementation, we compared the spot diagrams from dO and Zemax (considered the groundtruth) of all four configurations across various wavelengths and spatial positions. Figure 5 shows a selection of four spot diagrams for various positions across the field of view for the (mSP) configuration at a wavelength of $\lambda = 650$ nm (more spot diagrams are presented in the supplementary material, Figures 11 to 14). Although this configuration exhibits the strongest distortions and is thus the most challenging of the four considered configurations, the spot diagrams modeled with dO are similar to their Zemax counterparts. The spot diagrams from both implementations (dO and Zemax) are highly consistent, with Root-Mean-Square (RMS) spot size differences below 1.5 µm, well below the considered 10-µm pixel size, thereby ensuring adequate rendering resolution.

Differences observed in spot diagrams primarily arise from aperture handling which impacts ray distribution: dO assumes a telecentric system without an aperture stop, whereas Zemax includes an aperture object. Adding an aperture function to dO would require ray-direction calculations for aperture filling, but the benefits would be minimal given the minor discrepancies between both systems. Additionally, ray sampling methods differ: with dO we employ hexapolar sampling, while Zemax uses grid sampling for spot diagram determination.

Further validation was conducted by comparing distortions at various wavelengths and positions obtained with both dO and Zemax. Figure 6 shows the difference between Zemax and dO distortions across the field of view for extrema
<figure><img src="image_5.png" /><figcaption>Fig. 5: Spot diagrams obtained with the misaligned single prism configuration (mSP) at four positions in the field of view (positions denoted by the red points on the bottom left of each figure). Top: spot diagrams obtained with dO, bottom: spot diagrams obtained with Zemax. The dotted black circle corresponds to the RMS radius centered on the centroid of the spot diagrams.</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>Fig. 6: Maps of the difference between distortions computed with dO and Zemax, for the extrema and central wavelengths (450 nm, 520 nm, and 650 nm). Note the same colorscales across the 4 configurations.</figcaption></figure>

and central wavelengths. Results show that distortions are accurately modeled, with mean differences of only 1 µm across the three wavelengths, a tenth of the actual pixel size. Minor discrepancies in distortion maps between dO and Zemax also stem from their differing aperture treatments.

### D. Coded acquisition rendering

1) Forward model: To simulate the acquisition of an HSS $\mathbf{H}$ of dimension $H \times W \times K$ –along the vertical, horizontal and spectral directions– with a given 2D binary mask $M$ through

a SD-CASSI system, we compute the coded scene $\mathbf{H}^c$ as follows:

$$ \mathbf{H}^c(\cdot, \cdot, k) = \mathbf{H}(\cdot, \cdot, k) \odot \mathbf{M}(\cdot, \cdot), \forall k \in [0, K] \quad (9) $$

where $\odot$ is the element-wise product, and $k$ the index along the spectral dimension. $\mathbf{H}^c$ thus represents information at the entry of our optical system.

The SD-CASSI systems then form the image of a coded scene $\mathbf{H}^c$ through the following forward model. Each configuration will entail a unique PSF, based on its optics, that will alter the resulting acquisition. Since our configurations can use both aligned and misaligned prisms, they exhibit spectral dispersions both along $x$ and $y$ dimensions.

In the case of shift-invariant PSFs, the coded acquisition $\mathbf{A}(x, y)$ formed on the detector can be written as:

$$ \mathbf{A}(x, y) = \int \mathbf{H}^c(x - s_x(\lambda), y - s_y(\lambda), \lambda) \otimes p(x, y, \lambda) d\lambda \quad (10) $$

with $\mathbf{H}^c$ the coded scene, $s_x$ and $s_y$ the wavelength dependent dispersion along the $x$ and $y$ axes, $p(\cdot, \cdot, \lambda)$ the PSF at wavelength $\lambda$, and $\otimes$ represents the 2D spatial convolution. Note that when the system is perfectly aligned, $s_y = 0$. However, when accounting for distortions and aberrations, the PSFs are no longer stationary, and vary both with the position on the detector and with the wavelength. This results in a more realistic model [29] described in the following equation:

$$ \mathbf{A}(x, y) = \iiint \mathbf{H}^c(x', y', \lambda) \times p(x - x' - s_x(\lambda), y - y' - s_y(\lambda); x, y, \lambda) dx'dy'd\lambda \quad (11) $$

After spatial and spectral discretization (which requires some approximations), equation (11) writes:

$$ \mathbf{A}(i_d, j_d) = \sum_k \sum_{\substack{i', j' \\ \in \Omega_{i_d, j_d, k}}} \mathbf{H}^c(i', j', k) \times p(i_d - i' - s_x(k), j_d - j' - s_y(k); i_d, j_d, k) \quad (12) $$

where $\Omega_{i_d, j_d, k}$ represents the pixels of the scene that contribute to a sensor pixel $(i_d, j_d)$ (center position of a detector pixel) at the discrete spectral index $k$. Such a model is difficult to manage as it requires to know the varying PSFs for all discrete positions $(i_d, j_d)$ and all wavelengths $k$. One recent approach [29] is to tabulate estimated PSFs with a good-enough spatial and spectral sampling and build a spatially and spectrally varying model in order to approximate the PSFs at any position. However, this approach requires access to an existing prototype to estimate the PSFs.

Contrarily, here we focus on CASSI systems at the design stage, where a prototype is yet to be built. Therefore, we propose to use a ray-tracing approach similar to our rendering process. Instead of weighting the coded scene $\mathbf{H}^c$ at pixel location $(i', j')$ with an approximated PSF, we propose to launch $R$ rays backward from each pixel of the detector to the coded scene for each wavelength. The forward model corresponding to such a ray-tracing approach writes:

$$ \mathbf{A}(i_d, j_d) = \sum_k \frac{1}{|\Omega_{i_d, j_d, k}|} \sum_{r=1}^{R} \mathbf{H}^c(x_r(k), y_r(k), k) \quad (13) $$
<figure><img src="image_7.png" /><figcaption>Fig. 7: Description of the forward and backward ray-tracing (a single wavelength is shown for readability). (a) Forward ray-tracing, with a single chief-ray to create the mapping function (red) and with many rays to trace the spot diagrams (blue). (b) Backward ray-tracing with *R* rays, used to render acquisitions by finding contributing voxels in the scene for every detector pixel.</figcaption></figure>

where $(x_r(k), y_r(k))$ corresponds to the pixel position of a ray $r$ on the scene plane after being traced from the detector pixel $(i_d, j_d)$ at wavelength index $k$ (wavelength $\lambda$, see Figure 7) and $|\Omega_{i_d, j_d, k}|$ is the number of such rays that actually reach the scene plane — some rays might be blocked somewhere along the optical path. $H^c(x_r(k), y_r(k), k)$ is approximated by interpolation from the values of neighboring pixels in the scene at wavelength $k$. This equation is close to Eq. (12) since we estimate which scene voxels contribute to the rendered image on a given detector pixel $(i_d, j_d)$, but the weighting is deduced from the backpropagated rays instead of the PSF. It is worth noting that using backward ray-tracing to describe the forward propagation is a well established approach for rendering, although rarely used for HS scenes. This process enables the efficient computation of the acquisition $A$ directly from the simulator, without requiring a pre-existing prototype.

2) **Rendering process:** Our model of the four optical configurations enables HSS rendering through backward ray-tracing. Each spectral plane is rendered individually, and acquisition is performed by summing along the spectral dimension as described in II-D1. We render scenes with 28 spectral bands evenly distributed from 450 nm to 650 nm from the hyperspectral datasets CAVE [47] and KAIST [40], preprocessed as in [19]. Due to the systems' 830 µm nonlinear spectral spread on the detector and 10 µm spatial sampling, the data is upsampled to ensure accurate rendering over the 83 illuminated pixels (otherwise some pixels would not receive any signal, giving a spatially non-continuous rendered acquisition). This necessity is highlighted in Supplementary Figure 15, where we show the rendered acquisitions of a scene upsampled with different upsampling factors. The smallest integer upsampling factor $n$ so that $28 \times n \ge 83$ is $n = 3$: we upsample with $n = 4$ to alleviate missing signal errors that could occur due to Monte-Carlo sampling with a small amount of rays. The upsampling is done through linear interpolation. We therefore use these upsampled scenes in equation (9).

Rendered HSSs therefore contain $4 \times 28 = 112$ spectral bands. For a fast rendering of the 512×512×112 hyperspectral scenes, we limit ray tracing to $R = 20$ rays per pixel per wavelength (see equation (12)). However, the upsampling is performed solely for the rendering step and as explained in Section III, the algorithms reconstruct the original 28-bands

HSS. Following the modifications done to dO, rendering is performed sequentially, progressing backwards through each optical element and independently for each wavelength.

To further mitigate quantization errors from Monte-Carlo sampling arising from tracing a small amount of rays, the rendered planes are convolved with a smoothing kernel. We convolve each spectral plane with an Airy kernel, varying according to $\lambda$, whose diameter equals 2.5 pixels ($= 25$ µm) at 520 nm, adequately smoothing the rendering through convolution.

3) **Validation of the Rendering Method:** We validate our rendering method with a vertical, 1-pixel-wide, single-slit mask, for which the SD-CASSI systems emulate an imaging prism spectrometer. As both aligned configurations (AP) and (SP) disperse the spectrum of the imaged slit horizontally along the $x$ axis, the spectrum of each point along the slit can be directly read on the corresponding horizontal line on the detector. Since distortions mainly appear in the (SP) configuration, we will only consider this harder case.

The test scene selected from the CAVE dataset is shown in Figure 8-(a), together with the chosen slit position (dashed line). The slit is located at the center of the field of view, crossing 3 regions (green, orange and gray squares) with a constant spectrum. The acquisition shown in Figure 8-(b) clearly illustrate the spatial dispersion and the effects of distortion. Our aim is to have a fair comparison of the spectra extracted from the acquisition and the ground truth, retrieved from the HSS, as illustrated in Figure 8-(c). On one hand, to account both for the $y$-axis spread and the rendering with a small amount of rays, we average 40 rows of the acquisition in each region. On the other hand, the ground truth spectra are derived from the HSS, accounting for the dispersion model and the convolution with the Airy kernel used in the rendering process. Moreover, we upsample the ground truth spectra to 280 wavelengths to account for their continuity and to reduce the quantization errors that would otherwise appear in the comparison, with no physical meaning.

Results presented in Figure 8-(c) show a close match between the acquisitions (solid lines) and the ground truth spectra (dashed lines). Thus, our model effectively simulates a SD-CASSI system, enabling realistic acquisition simulations that align with those from a physical system.

We created a fork of dO including all our modifications that can be accessed at https://github.com/lpaillet-laas/DiffOptics.

Additionally, another repository accessible at https://github.com/lpaillet-laas/DiffCassiSim, contains this fork of dO together with all the code for the processing done in Section III.

### III. HYPERSPECTRAL CUBE RECONSTRUCTIONS

#### A. Overview

We demonstrate here that for the four considered configurations, a comparable amount of information is encoded in the acquisitions and that it can be retrieved by state-of-the-art approaches when the system model is known and considered, hence exhibiting the marginal impact of distortions and misalignments.

For this purpose, the hyperspectral cube reconstruction from coded acquisition is an ideal test case. It consists in
<figure><img src="image_8.png" /><figcaption>Fig. 8: Spectra extracted from a rendered single-slit acquisition with the (SP) configuration compared to ground truth spectra retrieved from the HSS. The spectrum of each pixel along the slit is horizontally spread along the rows of the detector. (a) RGB image of the scene and location of the slit (dashed line). (b) Rendered acquisition with the (SP) configuration. (c) Acquired spectra (solid lines) in the 3 regions with constant color (green, orange and gray) compared to their ground truth counterparts (dashed lines). For each curve, the bottom scale is the position on the detector (in pixel), the top one is the corresponding wavelength.</figcaption></figure>

reconstructing a scene **H** from a coded acquisition **A**. Deep learning methods [9], [40], [48] being now the best solution for HSC reconstruction, we apply five recent state-of-the-art algorithms, and assess the reconstruction quality with a series of usual metrics. The considered reconstruction algorithms are Deep Gaussian Scale Mixture Prior (DGSMP) [21], Mask-guided Spectral-wise Transformer (MST) [19], Degradation-Aware Unfolding Half-Shuffle Transformer (DAUHST) [22], Residual Degradation Learning Unfolding Framework (RD-LUF) [23], and Pixel Adaptive Deep Unfolding Transformer (PADUT) [24].

Our evaluation uses the two public HSS datasets CAVE [47] and KAIST [40]. The CAVE dataset includes 32 HSSs of size 512 × 512 × 31, while the KAIST dataset includes 30 HSSs of size 2704 × 3376 × 31. HSSs have been spectrally interpolated following [47] in order to be consistent with the wavelengths of the system used to acquire these HSSs. They thus contain 28 spectral bands ranging from 450 nm to 650 nm. We consider these spectral bands to be evenly distributed over this range. To fit the field of view (≈ 5 × 5 mm²) with a pixel size of 10 µm, each HSS in the datasets was cropped to reach a final size of 512 × 512 × 28. Following the setups of the referenced networks, models were trained on the CAVE dataset and tested on 10 scenes taken from the KAIST dataset.

## B. Spatio-Spectral Mapping and Reconstruction Initialization

Most networks leverage a priori information from the acquisition to reconstruct the hyperspectral cube. For perfectly aligned systems without distortions, a scene **H** with dimensions

$H \times W \times K$ is rendered onto an acquisition **A** of size $H \times (W + S)$, where $S$ denotes the spectral spread (in pixels) of the optical system. Defining the spectral spread at a specific wavelength index $k$ as $s(k)$, the typical network initialization **I** is given by:

$$ \mathbf{I}(\cdot, \cdot, k) = \mathbf{A}(\cdot, s(k) : W + s(k)), \forall k \in [0, K] \quad (14) $$

resulting in an initialized cube of dimensions $H \times W \times K$.

With negligible distortions, all information from the scene at wavelength index $k$ is indeed captured in a rectangular area of size $H \times W$ starting at the position $i_d = s(k)$. However, as shown in Figure 4 and in Figure 17 of the supplementary material, some of our configurations exhibit spatial distortions, so information from each wavelength is not represented in a rectangular area. In the misaligned configurations (mAP) and (mSP), the acquisition **A** also displays dispersion along the $y$ axis, resulting in an acquisition of dimensions $(H + S_x) \times (W + S_y)$, where $S_x$ denotes the spectral spread (in pixels) along the $x$ axis, and $S_y$ the spectral spread along the $y$ axis.

Knowing both the configurations and our model, we can define a mapping $f$ for each configuration such that:

$$ f(i_s, j_s, k) = (i_d, j_d) \quad (15) $$

where $(i_s, j_s, k)$ denotes a voxel of the scene and $(i_d, j_d)$ is the corresponding position on the detector. To estimate the one-to-one mapping described by equation (15), we forwardly ray trace a single ray parallel to the optical axis – a chief ray – for each position in the scene and each wavelength (red ray in Figure 7). This mapping estimates where the spectral intensity originating from the scene pixel $(i_s, j_s)$ is projected onto the detector, at a given wavelength. In contrast to simplified models that assume a regular, ordered spatial layout, this formulation can accommodate complex information arrangements.

Using the mapping $f$, for each wavelength $k$, we can accurately fit parts of the distorted acquisition into a rectangle of shape $H \times W$, thus initializing **I** as:

$$ \mathbf{I} = \mathbf{A} \circ f \quad (16) $$

This initialization ensures accurate computation, aligning with the model and the specific optical system in use.

The considered reconstruction algorithms contain a simplified propagation model to achieve better results than if they were purely data-driven. Therefore, these algorithms have been modified to incorporate the mapping function $f$ in the reconstruction process, as it allows the computation of both the direct and adjoint operators used in the unfolded iterative steps. This allows an accurate propagation model to be integrated in the reconstruction algorithm, faithful to how acquisitions are generated.

## C. Networks Implementation

The networks were adapted to accommodate our rendered acquisitions. Originally designed to reconstruct 256 × 256 × 28 HSCs, the networks were modified to reconstruct 512 × 512 × 28 HSCs required by our configurations. Additionally, most
<figure><img src="image_9.png" /><figcaption>Fig. 9: Workflow to reconstruct HSCs with a given optical system and a given reconstruction algorithm. The scene **H** is upsampled into **H**<sub>up</sub> and then acquired with the optical system and the mask **M** thanks to our rendering with dO. The acquisition is then mapped to an initialization using the model through *f*. This serves as an input to a reconstruction algorithm yielding **H**. The mapping function is also used in the algorithm.</figcaption></figure>

networks were previously hardcoded for simplified optical setups assuming linear spectral dispersion with a slope of 2 pixels (20 µm in our case) across 560 µm. However, our configurations exhibit unique non-linear spectral spread across 830 µm. The networks were therefore adapted with the mapping function *f* to process the acquisitions, as illustrated in Figure 9.

For all HSSs, a unique binary mask with a random opening ratio of 0.5 was used. Consistent with original methods, we use the Adam optimizer and both learning rates and scheduler schemes were maintained. Each network and configuration

was trained for 500 epochs, utilizing random cropping, flipping, and rotation for data augmentation. The batch size was set to 1, with gradient accumulation every 4 batches, and training conducted on NVIDIA RTX A100 and A6000 GPUs.

Training loss was calculated as the Root Mean Square Error (RMSE) between the ground truth HSSs and reconstructed HSCs. Evaluation metrics include RMSE, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Spectral Angle Mapper (SAM) [49]. The SAM metric allows to evaluate the spectral accuracy of the reconstructions.

### D. Results

<table><caption>TABLE I: Average RMSE, PSNR (in dB), SSIM and SAM on all test scenes, for each optical system and reconstruction algorithm. Bold numbers correspond to the best configuration for a given metric and algorithm.</caption><thead><tr><th></th><th></th><th>(AP)</th><th>(SP)</th><th>(mAP)</th><th>(mSP)</th></tr></thead><tbody><tr><td rowspan="5">RMSE ↓<br>(×10<sup>-3</sup>)</td><td>DGSMP</td><td><strong>31.3</strong></td><td>32.8</td><td>31.8</td><td>34.4</td></tr><tr><td>MST</td><td>27.3</td><td><strong>26.5</strong></td><td><strong>26.5</strong></td><td>27.0</td></tr><tr><td>DAUHST</td><td>20.7</td><td>19.6</td><td><strong>19.5</strong></td><td>20.3</td></tr><tr><td>RDLUF</td><td><strong>22.6</strong></td><td>24.1</td><td>23.8</td><td>23.7</td></tr><tr><td>PADUT</td><td>21.4</td><td><strong>19.6</strong></td><td>21.6</td><td>21.8</td></tr><tr><td rowspan="5">PSNR ↑</td><td>DGSMP</td><td><strong>30.6</strong></td><td>30.3</td><td>30.5</td><td>29.9</td></tr><tr><td>MST</td><td>31.9</td><td><strong>32.2</strong></td><td>32.1</td><td>32.0</td></tr><tr><td>DAUHST</td><td>34.4</td><td>34.7</td><td><strong>34.9</strong></td><td>34.4</td></tr><tr><td>RDLUF</td><td><strong>33.5</strong></td><td>32.9</td><td>33.0</td><td>33.0</td></tr><tr><td>PADUT</td><td>34.1</td><td><strong>34.7</strong></td><td>34.0</td><td>33.8</td></tr><tr><td rowspan="5">SSIM ↑<br>[0 - 1]</td><td>DGSMP</td><td><strong>0.892</strong></td><td>0.883</td><td>0.886</td><td>0.879</td></tr><tr><td>MST</td><td>0.910</td><td><strong>0.914</strong></td><td>0.912</td><td>0.911</td></tr><tr><td>DAUHST</td><td>0.942</td><td><strong>0.945</strong></td><td><strong>0.945</strong></td><td>0.939</td></tr><tr><td>RDLUF</td><td><strong>0.930</strong></td><td>0.922</td><td>0.926</td><td>0.926</td></tr><tr><td>PADUT</td><td>0.937</td><td><strong>0.943</strong></td><td>0.936</td><td>0.935</td></tr><tr><td rowspan="5">SAM ↓<br>[0 - 1]</td><td>DGSMP</td><td><strong>0.058</strong></td><td>0.062</td><td>0.059</td><td>0.065</td></tr><tr><td>MST</td><td>0.055</td><td><strong>0.052</strong></td><td>0.055</td><td>0.055</td></tr><tr><td>DAUHST</td><td>0.048</td><td>0.048</td><td><strong>0.047</strong></td><td>0.050</td></tr><tr><td>RDLUF</td><td><strong>0.053</strong></td><td>0.055</td><td>0.054</td><td><strong>0.053</strong></td></tr><tr><td>PADUT</td><td><strong>0.048</strong></td><td><strong>0.048</strong></td><td>0.049</td><td><strong>0.048</strong></td></tr></tbody></table>

As stated earlier, the networks were trained with our four configurations on the CAVE dataset and were then tested on the KAIST dataset. The quantitative results are presented in Table I. SAM has been normalized between 0 and 1 to be in the same range as SSIM. We compare the evaluation metrics across all four configurations for each state-of-the-art reconstruction algorithm. The evaluation metrics are slightly worse than those reported in corresponding papers. This occurs because we adapted the networks to process 512 × 512 × 28 HSCs, although they were originally designed to reconstruct 256 × 256 × 28 HSCs. Moreover, our spectral resolution leads to a spreading range of 83 pixels against the 54 usually considered for the simplified SD-CASSI models. Finally, we only passed through the preprocessed dataset [19] once per epoch. Thus, achieving the same performance as the reference papers might require either more training epochs or a deeper architecture to better capture small spatial variations and bigger spectral spreading. Some modifications to the learning procedures, or even to the algorithms themselves, could have
improved the results for each algorithm. However, as our aim is not to compare algorithms with each other but, for a given algorithm, to compare configurations with each other, such optimization is not crucial.

As seen in Table I, for a given reconstruction algorithm, all four evaluation metrics show no significant difference across all four configurations. For all reconstruction algorithms, the SSIM difference does not exceed 0.013, and the PSNR difference does not exceed 0.9 dB. For further details, the metrics for each scene can be seen in the supplementary materials (see Supplementary Tables IV to VIII). These non-significant differences indicate that the encoded information quality remains consistent across all four configurations, despite distortions and misalignments in some configurations. Moreover, the configuration that yields the best results varies depending on both the metric and the algorithm used, with each configuration performing best at least once. This means there is no single overall best configuration. This further validates that processing results mainly depend on the quality of the processing algorithm, rather than the optical system, provided the system is accurately modeled. Thus, we can relax certain constraints on optical design while achieving the same processing performance.

## E. Ablation Study

An ablation study was conducted to evaluate the impact of rendering accuracy and mapping on HSC reconstruction. The study was performed on two distortion-inducing configurations: (SP) and (mSP), and exclusively employed the PADUT algorithm, as it provided the highest reconstruction quality.

In the first experiment, we assessed the effect of incorrect rendering during the training phase. PADUT was trained using a simplified rendering, and reconstruction quality metrics were evaluated with our realistic rendering. The simplified model renders spectral planes using the (AP) system, which introduces negligible distortions and serves as the baseline for standard simplified frameworks [8], [11], [39]. To avoid bias in reconstruction, the spatio-spectral spreading *S* curve was set to match that of the realistic propagation model, minimizing minor differences between the systems (see Figure 3). Spatial shifts were thus applied to the rendered spectral planes to account for spatio-spectral spreading mismatches before summing them to form the acquisitions.

In the second experiment, the mapping function *f* was entirely removed from the workflow, meaning a simplified propagation model is used both during the training and the test. Indeed, only the horizontal spatio-spectral spreading characteristics of the configurations were used for initial reconstruction and during the algorithm's processing, to account for the correct dispersion and to prevent bias. This experiment isolates the influence of the mapping function on reconstruction quality. Both (SP) and (mSP) configurations were considered to account for distortions and misalignments, with the mapping excluded during training and testing.

Results for the first experiment are presented in Table II. They demonstrate that using different rendering models between the training and test phase significantly degrades reconstruction quality. However, using the same rendering method

<table><caption>TABLE II: Rendering ablation result, with PADUT algorithm.</caption><thead><tr><th>System</th><th>Training</th><th>Test</th><th>PSNR ↑</th><th>SSIM ↑</th></tr></thead><tbody><tr><td rowspan="3">(SP)</td><td>Simple</td><td>Simple</td><td>33.6</td><td>0.932</td></tr><tr><td>Simple</td><td>Real</td><td>22.2</td><td>0.730</td></tr><tr><td>Real</td><td>Real</td><td><b>34.7</b></td><td><b>0.943</b></td></tr><tr><td rowspan="3">(mSP)</td><td>Simple</td><td>Simple</td><td><b>34.3</b></td><td><b>0.937</b></td></tr><tr><td>Simple</td><td>Real</td><td>17.7</td><td>0.568</td></tr><tr><td>Real</td><td>Real</td><td>33.8</td><td>0.935</td></tr></tbody></table>

<table><caption>TABLE III: Mapping ablation result, with PADUT algorithm.</caption><thead><tr><th>System</th><th>Mapping</th><th>PSNR ↑</th><th>SSIM ↑</th></tr></thead><tbody><tr><td rowspan="2">(SP)</td><td>Without</td><td>33.6</td><td>0.936</td></tr><tr><td>With</td><td><b>34.7</b></td><td><b>0.943</b></td></tr><tr><td rowspan="2">(mSP)</td><td>Without</td><td>32.2</td><td>0.916</td></tr><tr><td>With</td><td><b>33.8</b></td><td><b>0.935</b></td></tr></tbody></table>

for both phases gives similar performances for all configurations, as seen in section III-D. This highlights the fact that reconstruction algorithms cannot be effectively trained on acquisitions rendered using simplified models with the expectation of robust performance when tested on systems employing realistic, non-simplified rendering processes, or on real acquisitions from CASSI prototypes.

In the second experiment (see Table III), the removal of the mapping function *f* from the reconstruction process only caused a slight deterioration in reconstruction quality. We hypothesize that the limited impact is due to the narrow spatio-spectral spread difference between consecutive wavelengths, which allows missing spatial information to be interpolated from neighboring wavelengths during training. Still, excluding a correct mapping during the training phase notably reduces the achievable quality metrics, thereby amplifying the performance gap relative to systems without distortions or misalignments.

Ultimately, rendering accuracy during both training and test has a much larger influence on reconstruction quality than the inclusion of a correct mapping function. However, a proper mapping still contributes to improved algorithm performance, providing a non-negligible enhancement in reconstruction accuracy. Both steps are therefore important to reach the best reconstruction quality with a given algorithm.

## F. Tolerance to misalignment

We further investigated the impact of misalignment on system performance by studying the evolution of reconstruction quality with increasing misalignment angle $\beta$. To this end, we trained the PADUT algorithm using (mSP) and (mAP) configurations with a misalignment $\beta$ increasing from 0° to 10°. We then compared the quality of the reconstruction (using PSNR metric) for two testing scenarios reported in Figure 10: using both training and mapping function corresponding to the actual misalignment $\beta$ (blue curve), using both training and mapping function at a fixed value of $\beta = 5°$ (orange curve).

Under the first scenario (blue curve), when both the training and mapping function match the correct misalignment $\beta$, the reconstruction is highly tolerant to misalignment for both
<figure><img src="image_10.png" /><figcaption>Fig. 10: Impact of misalignment angle $\beta$ (a) for (mSP), (b) for (mAP), using PADUT. In blue, the correct misalignment is used for training and mapping; in orange, for both training and mapping function a fixed misalignment of 5° is used.</figcaption></figure>

(mSP) and (mAP) configurations. Indeed, over the whole misalignment range, (mSP) exhibits a PSNR drop lower than 1.4 dB – thus barely significant – while (mAP) shows an almost flat PSNR curve. This shows that when using an accurate modeling of the image formation during both training and testing, high and consistent quality reconstruction can be achieved over a wide range of misalignments.

However, (mSP) and (mAP) have different behaviors for the second scenario (orange curve) where we consider a fixed reference misalignment of $\beta = 5°$ for training and mapping. While they both show significantly lower PSNR when moving away from the reference misalignment, the tolerance is much narrower and the performance drop higher for (mSP) than (mAP). Indeed, for a change of misalignment of around $\pm 0.5°$ around 5°, the PSNR of (mSP) decreases by $\approx 12$ dB while that of (mAP) stays almost unchanged (the biggest change in that range being lower than 1 dB).

These findings indicate that for both configurations, using an accurate mapping function and appropriate training can largely improve the tolerance to misalignment and yield high reconstruction quality over a wide range of misalignment, even for simple configurations like (mSP).

## IV. CONCLUSION

We focused on the design and performance evaluation of Coded Aperture Snapshot Spectral Imaging (CASSI) systems. Our work aimed to bridge the gap between optical hardware

and computational processing by providing realistic simulations and analyses that consider the complexities of actual optical setups.

Our main contribution was to show that even with simple optical systems, significant distortions and large misalignments, modern algorithms can reconstruct high quality hyperspectral scenes, provided that the image formation model used for training and reconstruction is realistic enough.

In order to reach that goal, our first contribution was the implementation of coded aperture hyperspectral optical systems within a differentiable ray-tracing framework. This enables to render synthetic coded hyperspectral images that accurately incorporate optical distortions and aberrations. By leveraging this framework, we could simulate four optical designs more precisely than with simplified mathematical models. Secondly, a realistic propagation model was utilized to map 2D coded measurements to estimations of hyperspectral cubes, serving both as the initialization and throughout the reconstruction process. This approach allowed for improved reconstruction by accounting for the exact optical characteristics of the configuration, including geometric distortions and spectral dispersion. This workflow can be generalized to a great variety of systems with our framework, given the propagation model is known. Moreover, when working with a prototype, one can also derive the mapping function from acquired experimental PSFs. Thirdly, we demonstrated that geometric distortions and misalignments in CASSI systems have a marginal impact on reconstruction performance. We conclude that the same amount of information is transmitted regardless of the system used.

Looking forward, this work yields the comparison of different CASSI system performances using information theory measures, facilitating more informed design choices [50]–[52], with no need for a processing algorithm to evaluate the design. It additionally opens several avenues for future research. Ideal thin lenses could be replaced by a surrogate model [53] trained with the help of Zemax blackbox files to have access to a wider range of lenses. The differentiability of the ray-tracing simulator can be exploited for optimizing coded apertures, potentially leading to designs that maximize information capture or minimize reconstruction error. Dynamic mask designs could also be achieved, accurately adapting to a scene in order to process several coded acquisitions and reach better information acquisition. Finally, this work paves the way for end-to-end optimization of CASSI systems, integrating both optical design and computational algorithms to achieve optimal performance in co-design computational imaging.

## ACKNOWLEDGMENTS

This project has received financial support from the CNRS through the MITI interdisciplinary programs.

## REFERENCES

[1] Y. Garini, I. T. Young, and G. McNamara, "Spectral imaging: principles and applications," *Cytometry A*, vol. 69, no. 8, pp. 735-747, Aug. 2006.

[2] X. Yuan, D. J. Brady, and A. K. Katsaggelos, "Snapshot compressive imaging: Theory, algorithms, and applications," *IEEE Signal Processing Magazine*, vol. 38, no. 2, pp. 65-88, 2021.
[3] A. Wagadarikar, R. John, R. Willett, and D. Brady, "Single disperser design for coded aperture snapshot spectral imaging," Appl. Opt., vol. 47, no. 10, pp. B44-B51, Apr 2008. [Online]. Available: https://opg.optica.org/ao/abstract.cfm?URI=ao-47-10-B44

[4] G. R. Arce, D. J. Brady, L. Carin, H. Arguello, and D. S. Kittle, "Compressive coded aperture spectral imaging: An introduction," *IEEE Signal Processing Magazine*, vol. 31, no. 1, pp. 105-115, 2014.

[5] X. Cao, T. Yue, X. Lin, S. Lin, X. Yuan, Q. Dai, L. Carin, and D. J. Brady, "Computational snapshot multispectral cameras: Toward dynamic capture of the spectral world," *IEEE Signal Processing Magazine*, vol. 33, no. 5, pp. 95-108, 2016.

[6] M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, "Single-shot compressive spectral imaging with a dual-disperser architecture," *Opt. Express*, vol. 15, no. 21, pp. 14013-14027, Oct 2007. [Online]. Available: https://opg.optica.org/oe/abstract.cfm?URI=oe-15-21-14013

[7] D. Donoho, "Compressed sensing," *IEEE Transactions on Information Theory*, vol. 52, no. 4, pp. 1289-1306, 2006.

[8] A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, "Video rate spectral imaging using a coded aperture snapshot spectral imager," *Opt. Express*, vol. 17, no. 8, pp. 6368-6388, Apr 2009. [Online]. Available: https://opg.optica.org/ao/abstract.cfm?URI=oe-17-8-6368

[9] L. Wang, T. Zhang, Y. Fu, and H. Huang, "HyperReconNet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging," *IEEE Trans. Image Process.*, vol. 28, no. 5, pp. 2257-2270, Nov. 2018.

[10] I. Ardi, H. Carfantan, S. Lacroix, and A. Monmayrant, "Fast hyperspectral cube reconstruction for a double disperser imager," in *2018 26th European Signal Processing Conference (EUSIPCO)*. IEEE, sep 2018.

[11] D. Kittle, K. Choi, A. Wagadarikar, and D. J. Brady, "Multiframe image estimation for coded aperture snapshot spectral imagers," *Appl. Opt.*, vol. 49, no. 36, pp. 6824-6833, Dec 2010. [Online]. Available: https://opg.optica.org/ao/abstract.cfm?URI=ao-49-36-6824

[12] Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, "Rank minimization for snapshot compressive imaging," *IEEE Transactions on Pattern Analysis and Machine Intelligence*, vol. 41, no. 12, pp. 2990-3006, 2019.

[13] X. Yuan, "Generalized alternating projection based total variation minimization for compressive sensing," in *2016 IEEE International Conference on Image Processing (ICIP)*, 2016, pp. 2539-2543.

[14] Y. Fu, Y. Zheng, I. Sato, and Y. Sato, "Exploiting spectral-spatial correlation for coded hyperspectral image restoration," in *2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2016, pp. 3727-3736.

[15] L. Huang, R. Luo, X. Liu, and X. Hao, "Spectral imaging with deep learning," *Light: Science & Applications*, vol. 11, no. 1, p. 61, Mar 2022. [Online]. Available: https://doi.org/10.1038/s41377-022-00743-6

[16] X. Miao, X. Yuan, Y. Pu, and V. Athitsos, "Lambda-Net: Reconstruct Hyperspectral Images From a Snapshot Measurement," in *2019 IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019, pp. 4058-4068.

[17] Z. Meng, J. Ma, and X. Yuan, "End-to-end low cost compressive spectral imaging with spatial-spectral self-attention," in *Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII*. Berlin, Heidelberg: Springer-Verlag, 2020, p. 187-204. [Online]. Available: https://doi.org/10.1007/978-3-030-58592-1_12

[18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. U. Kaiser, and I. Polosukhin, "Attention is all you need," in *Advances in Neural Information Processing Systems*, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

[19] Y. Cai, J. Lin, X. Hu, H. Wang, X. Yuan, Y. Zhang, R. Timofte, and L. Van Gool, "Mask-guided spectral-wise transformer for efficient hyperspectral image reconstruction," in *2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. Los Alamitos, CA, USA: IEEE Computer Society, Jun 2022, pp. 17481-17490. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.01698

[20] ——, "Coarse-to-fine sparse transformer for hyperspectral image reconstruction," in *Computer Vision - ECCV 2022*, S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, Eds. Cham: Springer Nature Switzerland, 2022, pp. 686-704.

[21] T. Huang, W. Dong, X. Yuan, J. Wu, and G. Shi, "Deep gaussian scale mixture prior for spectral compressive imaging," in *2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. Los Alamitos, CA, USA: IEEE Computer Society, Jun 2021, pp. 16211-16220. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.01595

[22] Y. Cai, J. Lin, H. Wang, X. Yuan, H. Ding, Y. Zhang, R. Timofte, and L. V. Gool, "Degradation-aware unfolding half-shuffle transformer for spectral compressive imaging," in *Advances in Neural Information Processing Systems*, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, Eds., vol. 35. Curran Associates, Inc., 2022, pp. 37749-37761. [Online]. Available: https://proceedings.neurips.cc/paper_files/paper/2022/file/f621c2ead473ca36763696b712ffda01-Paper-Conference.pdf

[23] Y. Dong, D. Gao, T. Qiu, Y. Li, M. Yang, and G. Shi, "Residual degradation learning unfolding framework with mixing priors across spectral and spatial for compressive spectral imaging," in *2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. Los Alamitos, CA, USA: IEEE Computer Society, Jun 2023, pp. 22262-22271. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/CVPR52729.2023.02132

[24] M. Li, Y. Fu, J. Liu, and Y. Zhang, "Pixel adaptive deep unfolding transformer for hyperspectral image reconstruction," in *2023 IEEE/CVF International Conference on Computer Vision (ICCV)*. Los Alamitos, CA, USA: IEEE Computer Society, Oct 2023, pp. 12913-12922. [Online]. Available: https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.01191

[25] X. Lin, Y. Liu, J. Wu, and Q. Dai, "Spatial-spectral encoded compressive hyperspectral imaging," *ACM Transactions on Graphics*, vol. 33, no. 6, pp. 1-11, Nov. 2014.

[26] S. McGregor, S. Lacroix, and A. Monmayrant, "Adaptive hyperspectral imager: design, modeling, and control," *Journal of Optics*, vol. 17, no. 8, p. 085607, 2015. [Online]. Available: http://stacks.iop.org/2040-8986/17/i=8/a=085607

[27] Y. Saita, D. Shimoyama, R. Takahashi, and T. Nomura, "Single-shot compressive hyperspectral imaging with dispersed andundispersed light using a generally available grating," *Appl. Opt.*, vol. 61, no. 5, pp. 1106-1111, Feb 2022. [Online]. Available: https://opg.optica.org/ao/abstract.cfm?URI=ao-61-5-1106

[28] L. Lin, G. Wetzstein, Y. Liu, and Q. Dai, "Dual-coded compressive hyperspectral imaging," *Opt. Lett.*, vol. 39, no. 7, pp. 2044-2047, Apr 2014. [Online]. Available: https://opg.optica.org/ol/abstract.cfm?URI=ol-39-7-2044

[29] L. Song, L. Wang, M. H. Kim, and H. Huang, "High-accuracy image formation model for coded aperture snapshot spectral imaging," *IEEE Transactions on Computational Imaging*, vol. 8, pp. 188-200, 2022.

[30] L. Wang, Z. Xiong, D. Gao, G. Shi, W. Zeng, and F. Wu, "High-speed hyperspectral video acquisition with a dual-camera architecture," in *2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2015, pp. 4942-4950.

[31] L. Wang, Z. Xiong, D. Gao, G. Shi, and F. Wu, "Dual-camera design for coded aperture snapshot spectral imaging," *Appl. Opt.*, vol. 54, no. 4, pp. 848-858, Feb. 2015.

[32] Z. Xiong, L. Wang, H. Li, D. Liu, and F. Wu, "Snapshot hyperspectral light field imaging," in *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 2017, pp. 6873-6881.

[33] J. Feng, X. Fang, X. Cao, C. Ma, Q. Dai, H. Zhu, and Y. Wang, "Advanced hyperspectral video imaging system using amici prism," *Opt. Express*, vol. 22, no. 16, pp. 19348-19356, Aug 2014. [Online]. Available: https://opg.optica.org/oe/abstract.cfm?URI=oe-22-16-19348

[34] A. Rouxel, A. Monmayrant, S. Lacroix, H. Camon, and S. Lopez, "Accurate ray-tracing optical model for coded aperture spectral snapshot imagers," *Applied Optics*, vol. 63, no. 7, p. 1828, Feb. 2024.

[35] Q. Sun, C. Wang, Q. Fu, X. Dun, and W. Heidrich, "End-to-end complex lens design with differentiate ray tracing," *ACM Transactions on Graphics*, vol. 40, no. 4, pp. 1-13, Jul. 2021.

[36] E. Tseng, A. Mosleh, F. Mannan, K. St-Arnaud, A. Sharma, Y. Peng, A. Braun, D. Nowrouzezahri, J.-F. Lalonde, and F. Heide, "Differentiable compound optics and processing pipeline optimization for end-to-end camera design," *ACM Trans. Graph.*, vol. 40, no. 2, Jun. 2021. [Online]. Available: https://doi.org/10.1145/3446791

[37] C. Wang, N. Chen, and W. Heidrich, "dO: A Differentiable Engine for Deep Lens Design of Computational Imaging Systems," *IEEE Transactions on Computational Imaging*, vol. 8, pp. 905-916, 2022.

[38] N. Diaz, J. Ramirez, E. Vera, and H. Arguello, "Adaptive multisensor acquisition via spatial contextual information for compressive spectral image classification," *IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing*, vol. 14, pp. 9254-9266, 2021.

[39] H. Arguello, H. Rueda, Y. Wu, D. W. Pather, and G. R. Arce, "Higher-order computational model for coded aperture spectral imaging," *Appl.
Opt., vol. 52, no. 10, pp. D12-D21, Apr 2013. [Online]. Available: https://opg.optica.org/ao/abstract.cfm?URI=ao-52-10-D12

[40] I. Choi, D. S. Jeon, G. Nam, D. Gutierrez, and M. H. Kim, "High-quality hyperspectral reconstruction using a spectral prior," *ACM Trans. Graph.*, vol. 36, no. 6, Nov. 2017. [Online]. Available: https://doi.org/10.1145/3130800.3130810

[41] L. Paillet, A. Rouxel, H. Carfantan, A. Monmayrant, and S. Lacroix, "Differentiable chief-ray tracing simulator for coded-aperture spectral imaging," in *Optica Imaging Congress 2024 (3D, AOMS, COSI, ISA, pcAOP)*, ser. COSI, vol. 7. Optica Publishing Group, 2024, p. CTh4B.2.

[42] A. Rouxel and L. Paillet, "SIMCA," https://github.com/a-rouxel/simca, 2023.

[43] G. Côté, F. Mannan, S. Thibault, J.-F. Lalonde, and F. Heide, "The differentiable lens: Compound lens search over glass surfaces and materials for object detection," in *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, June 2023.

[44] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," 2014.

[45] R. Zhao, C. Yang, R. T. Smith, and L. Gao, "Coded aperture snapshot spectral imaging fundus camera," *Scientific Reports*, vol. 13, no. 1, p. 12007, Jul 2023. [Online]. Available: https://doi.org/10.1038/s41598-023-39117-2

[46] M. Zhang, L. Wang, and H. Huang, "Sub-pixel dispersion model for coded aperture snapshot spectral imaging," *IEEE Transactions on Computational Imaging*, vol. 9, pp. 1188-1200, 2023.

[47] F. Yasuma, T. Mitsunaga, D. Iso, and S. Nayar, "Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum," *IEEE Transactions on Image Processing*, vol. 99, Mar 2010.

[48] L. Wang, C. Sun, Y. Fu, M. H. Kim, and H. Huang, "Hyperspectral image reconstruction using a deep spatial-spectral prior," in *2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2019, pp. 8024-8033.

[49] R. H. Yuhas, A. F. H. Goetz, and J. W. Boardman, "Discrimination among semi-arid landscape endmembers using the spectral angle mapper (sam) algorithm," *Summaries of the Third Annual JPL Airborne Geoscience Workshop*, vol. 1, June 1992.

[50] E. H. Linfoot, "Information theory and optical images," *J. Opt. Soc. Am.*, vol. 45, no. 10, pp. 808-819, Oct 1955. [Online]. Available: https://opg.optica.org/abstract.cfm?URI=josa-45-10-808

[51] J. O'Sullivan, R. Blahut, and D. Snyder, "Information-theoretic image formation," *IEEE Transactions on Information Theory*, vol. 44, no. 6, pp. 2094-2123, 1998.

[52] A. Ashok and M. A. Neifeld, "Information-based analysis of simple incoherent imaging systems," *Opt. Express*, vol. 11, no. 18, pp. 2153-2162, Sep 2003. [Online]. Available: https://opg.optica.org/oe/abstract.cfm?URI=oe-11-18-2153

[53] T. Goossens, Z. Lyu, J. Ko, G. C. Wan, J. Farrell, and B. Wandell, "Ray-transfer functions for camera simulation of 3d scenes with hidden lens design," *Opt. Express*, vol. 30, no. 13, pp. 24031-24047, Jun 2022. [Online]. Available: https://opg.optica.org/oe/abstract.cfm?URI=oe-30-13-24031

V. BIOGRAPHY SECTION

Léo Paillet photograph and biography not available at the time of publication.

Antoine Rouxel photograph and biography not available at the time of publication.

Hervé Carfantan photograph and biography not available at the time of publication.

Simon Lacroix photograph and biography not available at the time of publication.

Antoine Monmayrant photograph and biography not available at the time of publication.