# Learning flat optics for extended depth of field microscopy imaging

Ipek Anil Atalay Appak, Erdem Sahin, Christine Guillemot, Humeyra Caglayan

► To cite this version:

Ipek Anil Atalay Appak, Erdem Sahin, Christine Guillemot, Humeyra Caglayan. Learning flat optics for extended depth of field microscopy imaging. Nanophotonics, 2023, pp.1-10. 10.1515/nanoph-2023-0321. hal-04184173

HAL Id: hal-04184173

https://hal.science/hal-04184173v1

Submitted on 21 Aug 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

Research Article

Ipek Anil Atalay Appak, Erdem Sahin, Christine Guillemot, and Humeyra Caglayan*

# Learning flat optics for extended depth of field microscopy imaging

https://doi.org/10.1515/sample-YYYY-XXXX

Received Month DD, YYYY; revised Month DD, YYYY; accepted Month DD, YYYY

**Abstract:** Conventional microscopy systems have limited depth of field, which often necessitates depth scanning techniques hindered by light scattering. Various techniques have been developed to address this challenge, but they have limited extended depth of field (EDOF) capabilities. To overcome this challenge, this study proposes an end-to-end optimization framework for building a computational EDOF microscope that combines a 4f microscopy optical setup incorporating a learned optics at the Fourier plane and a post-processing deblurring neural network. Utilizing the end-to-end differentiable model, we present a systematic design methodology for computational EDOF microscopy based on the specific visualization requirements of the sample under examination. In particular, we demonstrate that the metasurface optics provides key advantages for extreme EDOF imaging conditions, where the extended DOF range is well beyond what is demonstrated in state of the art, achieving superior EDOF performance.

**Keywords:** microscopy imaging, extended depth of field, metasurfaces, diffractive optics, end-to-end learning

## 1 Introduction

A conventional imaging system can produce sharp images for objects within the depth of field (DOF), which is the range around the focused depth of the scene. The DOF coverage is inversely proportional to the numerical aperture (NA), i.e., a smaller NA leads to a larger DOF. Although specific applications may require smaller DOFs, a larger DOF is often preferable to obtain sharp images of objects at varying depths. In particular, microscopy requires high NA objectives to capture precise details surrounding the focused depth of an object. However, due to the shallow DOF of high NA imaging systems, microscopic imaging systems often use depth scanning techniques to cover the entire depth range of interest, typically much larger than the imaging system DOF [20]. Depth scanning techniques are often insufficient due to the light scattering from objects outside the intended image plane, resulting in artifacts or severe noise. Acquiring cross-sectional data using this approach requires sweeping a focused point across the entire sample, which inherently imposes temporal limitations on the frame rate and prevents snapshot acquisition. To address these challenges, various techniques have been developed, such as decoupled illumination and detection in light-sheet microscopy [10], dynamic remote focusing [25, 34], and spatial and spectral multiplexing [1, 9]. Despite their potential advantages, these methods often necessitate a specialized and intricate optical configuration, which may render them both costly and difficult to implement in microscopy applications. In addition, computational approaches such as Fourier ptychographic microscopy demonstrated extended depth of field (EDOF) imaging capabilities [38]. However, this method relies on image reconstruction that assumes a thin sample illuminated by oblique plane waves, rendering it unsuitable for clinical fluorescence imaging applications.

<footnote>Ipek Anil Atalay Appak, Humeyra Caglayan, Faculty of Engineering and Natural Science, Photonics, Tampere University,
33720 Tampere, Finland, e-mail: anil.atalay@tuni.fi, humeyra.caglayan@tuni.fi</footnote>

<footnote>Erdem Sahin, Faculty of Information Technology and Communication Sciences, Tampere University, 33720 Tampere,
Finland, erdem.sahin@tuni.fi</footnote>

<footnote>Ipek Anil Atalay Appak, Christine Guillemot, INRIA Rennes - Bretagne Atlantique, France, christine.Guillemot@inria.fr</footnote>
The integration of wavefront encoding with computational reconstruction methods offers a cost-effective and efficient approach to improve EDOF imaging performance. Specifically, computational reconstruction methods, such as deep learning-based approaches and iterative optimization algorithms, have demonstrated substantial advancements in enhancing the EDOF imaging performance by effectively mitigating defocus blur, noise, and artifacts [2, 13, 17, 27, 33]. These methods leverage the optimization of both optical components and the reconstruction algorithm, enabling the system to enhance the EDOF performance effectively. End-to-end optimization frameworks, which jointly optimize the optical design and the associated reconstruction algorithms, have emerged as a powerful tool for addressing the EDOF challenge. This approach allows the system to learn and adapt to specific imaging requirements, thereby improving overall performance and enabling better control over trade-offs in resolution, noise, and depth range [27].

Wavefront coding involves the use of a transparent phase element, such as a diffractive optical element (DOE) or a free-form refractive lens, placed at the aperture plane [6, 7, 16, 26, 28, 29, 37]. The main objective of EDOF wavefront coding is to achieve a depth-invariant point spread function (PSF) while preserving information at all spatial frequencies. However, wavefront coding possesses limited EDOF capabilities, potentially due to constraints in the space-bandwidth product (SBP), which determines the information content captured by the imaging system. In comparison, metasurfaces, which are ultra-thin meta-optics composed of subwavelength nano-antennas, offer increased design flexibility and a superior SBP compared to DOEs [12, 31]. This advantage can be attributed to the rich modal characteristics of meta-optical scatterers, enabling multifunctional capabilities beyond traditional DOEs, encompassing polarization, frequency, and angle multiplexing [8, 18, 22, 23]. Consequently, metasurfaces exhibit a greater potential for addressing the EDOF microscopy challenge more effectively than conventional DOEs, with researchers having already leveraged their benefits in various applications such as flat optics for imaging [19, 35], polarization control [3], and holography [39]. Despite the promising capabilities of meta-optics, current EDOF metasurface imaging methodologies exhibit a limited depth range for EDOF imaging [4, 5, 11, 36].

In this study, we propose an end-to-end optimization framework designed for acquiring high-resolution images across an extensive DOF range within a microscopy system. The optics and post-processing algorithm are modeled as parts of the end-to-end differentiable computational image acquisition system, allowing for simultaneously optimizing both components. Our computational EDOF microscope employs a hybrid approach that combines a 4f microscopy optical setup with a learned wavefront modulating optical element at the Fourier plane. We explore metasurfaces and (conventional) DOEs for implementing such novel design modulation. The encoded image acquired at the sensor is post-processed by a convolutional neural network (CNN) that implements deblurring to achieve an EDOF image of the specimen. The optimization of the designated learning parameters for the optics, i.e., the phase modulation function and the deblurring CNN (D-CNN), is grounded in an end-to-end learning procedure employing a dataset of sharp images. This methodology emphasizes the importance of carefully outlining sampling requirements for various depth-of-field targets in order to achieve optimal imaging results. As a result, our systematic design methodology significantly outperforms existing state-of-the-art EDOF imaging techniques in terms of image quality and depth range.

## 2 Method

The EDOF microscopy problem addresses the difficulty of capturing high-resolution images across an extensive DOF range within a microscopy system. The objective of resolving this problem is to improve the imaging performance of the system by expanding its DOF, allowing the acquisition of sharp images over a larger depth range without necessitating mechanical focus adjustments. Figure 1 presents the proposed 4f system designed to address the EDOF microscopy problem, which consists of an optical module, a sensor, and a subsequent deblurring module for post-processing. The optical module employs a 4f imaging configuration that incorporates a phase coding mask at the aperture position. This mask, an optical element crafted to modulate incident light, introduces phase shifts across various regions through a transparent material with
<figure><img src="image_3.png" /><figcaption>Fig. 1: A 4f system computational EDOF microscope model that combines the optical module, sensor, and D-CNN.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: Overall representation of the end-to-end framework for joint optimization of the phase coding mask and the D-CNN. The end-to-end framework consists of an optical layer that simulates image formation for the learned optics, and the deblurring layer employs U-Net architecture to reconstruct in-focus images within the desired DOF range.</figcaption></figure>

spatially varying geometry. The purpose of using phase coding mask is to manipulate the optical wavefront, enabling a specific PSF or other desired properties within the optical system. In addressing this problem, the 4f system components remain fixed while optimizing the EDOF microscopy imaging system by learning the spatial distribution of the phase coding mask and the D-CNN weights for the targeted DOF ranges. Consequently, the objective is to achieve a defocus invariant PSF within the optical system, resulting in improved EDOF microscopy imaging performance.

Figure 2 depicts the proposed end-to-end learning procedure for the EDOF microscopy problem. This process accepts two inputs as high-resolution image patches and the predetermined depth range. The architecture of the framework consists of two main components: an optical layer and a D-CNN layer. During the training phase, the sensor image is generated through the optical layer, which involves simulating the image acquisition process based on depth value and the input image. The resulting sensor image is then fed into the D-CNN, which estimates the sharp, deblurred image as its output. The backpropagation algorithm is used to update the spatial phase distribution of the phase coding mask and the weights of the D-CNN at each iteration of the learning procedure, thus optimizing the parameters of the EDOF microscope in an end-to-end manner. The optimized phase coding mask is realized as a meta-optic element (metalens) or DOE, depending on the spatial phase distribution and sampling. After training is completed for the targeted DOF range, defocus invariant optical element and D-CNN model are obtained in the 4f system computational EDOF microscope model, shown in Figure 2. Further details regarding the optical and D-CNN layers are provided below.

## 2.1 Optical Layer

The optical layer is a computational module that operates based on the principles of wave optics and performs sensor image formation. In the following, we discuss sensor image formation by employing wave optics in particular and characterize the parameters of the EDOF microscope for the targeted DOF range for a 4f shift-invariant imaging system as illustrated in Figure 1. Here, the specimen is illuminated by
a monochromatic, spatially incoherent light source. The spatial coordinates in the Fourier and sensor planes are denoted as $(x_1, y_1)$ and $(x_2, y_2)$, respectively. A transparent biological specimen in the system can be represented as a stack of 2D images corresponding to a fixed scene depth where a slice of such a stack corresponds to $I_z(x_2, y_2)$ at the scene depth z. The contribution of such a slice to the sensor image, $I_z^s(x_2, y_2)$, is determined by convolution with the depth-dependent PSF, $h_z(x_2, y_2)$:

$$I_z^s(x_2, y_2) = I_z(x_2, y_2) * h_z(x_2, y_2). \qquad (1)$$

The final image captured at the sensor, $I^s(x_2, y_2)$, is then determined by integrating $I_z^s(x_2, y_2)$ over all depth values possible in the scene, considering sensor noise:

$$I^s(x_2, y_2) = \int I_z^s(x_2, y_2) dz + \eta_s, \qquad (2)$$

where $\eta_s \approx N(0, \sigma_s^2)$ is zero-mean Gaussian noise with standard deviation $\sigma_s$. Considering Eq.s 1 and 2, the recovery of a sharp image directly depends on $h_z(x_2, y_2)$. The PSF on the sensor plane can be modeled using Fourier optics as the square of the Fourier transform of the generalized pupil function:

$$h_z(x_2, y_2) = |F\{P(x_1, y_1)\}|^2, \qquad (3)$$

where $F\{\cdot\}$ denotes the Fourier transform operator and $P(x_1, y_1)$ is the pupil function, which describes the relative amplitude and phase changes of the wavefront at the Fourier plane:

$$P(x_1, y_1) = A(x_1, y_1)e^{i\phi(x_1, y_1)}. \qquad (4)$$

The pupil can be modulated using a phase coding mask introducing a phase term $\phi^M(x_1, y_1)$ to enhance the defocus invariance of the PSF in the targeted DOF range. The resultant phase term becomes:

$$\phi_z(x_1, y_1) = \phi_z^{DF}(x_1, y_1) + \phi^M(x_1, y_1). \qquad (5)$$

The defocus aberration due to the mismatch between in-focus depth $z_0$ and the actual depth z of a scene point is

$$\phi_z^{DF}(x_1, y_1) = \psi_z \frac{x_1^2 + y_1^2}{r^2}, \qquad (6)$$

where $\psi_z = \frac{\pi}{\lambda} \left(\frac{1}{z} - \frac{1}{z_0}\right) r^2$ is the defocus coefficient and r is the radius of the pupil. To represent all spatial frequencies $k_x, k_y$ supported by the objective lens on the pupil plane, the pupil size must be chosen using

$$r \ge \frac{f_1}{k} \sqrt{k_x^2 + k_y^2}, \qquad (7)$$

where $f_1$ is the focal length of the tube lens, and k is the wave number. The frequency support of the PSF, in particular, determines the reconstruction quality. To avoid aliasing due to undersampling of the defocused pupil function, which would otherwise result in a miscalculated PSF, the minimum sampling rate of the simulated phase coding mask must be determined. The sampling rate $\Delta s$ for a given depth range should satisfy [2]

$$\Delta s \le \frac{r\pi}{8\psi_{max}}, \qquad (8)$$

where $\psi_{max} = \max(|\psi_{z-}|, |\psi_{z+}|)$, the maximum defocus value within the scene. At the same time, we use such a relation as a guide in choosing the search space for the phase modulation function of the optical element, matching its degree of freedom with the complexity of the EDOF imaging problem at hand. This is observed to have a significant help in the end-to-end learning process, i.e., its convergence to the optimal solution. The optimized optical element can be realized as a metalens or DOE according to the calculated maximum $\Delta s$ for the targeted depth range.

During training, the optical layer optimizes depth-dependent PSFs in order to achieve depth independence by minimizing the impact of the defocus phase through the utilization of learned phase coding mask elements. For each iteration within the forward pass, this optimization is achieved by calculating Eq. 3 and optimizing
<figure><img src="image_5.png" /><figcaption>Fig. 3: Design and simulation of imaging metalenses. The metalenses are made up of GaN nanopillars with a refractive index of 2.42, where the thickness (Δh), sampling (Δs), and radius (Δr) are the design parameters. (a) Simulation of the nanopillars' transmission amplitude and phase response via FDTD. (b) Illustration of the full metalens design, transforming the spatial phase profile into a spatial radius (of nanopillar) distribution with respect to the target phase.</figcaption></figure>

the $\phi^M(x_1, y_1)$ phase modulation term. For both optics, the amplitude $A(x_1, y_1)$ of the pupil function is kept constant within the aperture diameter and is modeled as a circular function. The camera parameters $f_1$, r, and Δs are predefined as layer parameters. r and Δs is calculated for the selected objective lens and targeted depth range using Eq. 7 and 8, respectively.

For the DOE design, the phase modulation can be controlled through unit cell height. Specifically, the phase shift is given by the equation:

$$ \phi^M(x_1, y_1) = k(n - 1)h(x_1, y_1), \qquad (9) $$

where n denotes the refractive index of the DOE material which is 1.5, specifically at the design wavelength of 550 nm within the green spectrum. The DOE is assumed to be a lossless, therefore the amplitude is kept fixed as the circular function within the aperture diameter.

For the metalens design, phase accumulation is achieved through the waveguiding effect [14], whereby the height of the nanopillars is selected to provide $2\pi$ phase coverage across a range of radii. While the smallest possible diameter is primarily limited by fabrication constraints, the largest diameter is 50 nm smaller than Δs, as set by Eq. 8. To ensure high efficiency, the nanopillar height Δh is optimized at the 550 nm design wavelength. The phase and transmission responses are simulated via a finite-difference time-domain (FDTD) analysis, which involves varying the radius of the Gallium nitride (GaN) nanopillar on a Sapphire glass substrate. As depicted in Figure 3(a), the analysis results in full phase coverage (0-2π) with high transmission (overall greater than 83%) for 550 nm.

Irrespective of the desired target phase, designing a metalens involves converting a spatial phase profile into a corresponding spatial radius distribution. A resulting $\phi^M(x_1, y_1)$ phase profile, generated from the end-to-end framework, is illustrated in Figure 3(b). This profile is then translated into the full metalens, taking into account the simulated phase and radius response. This approach enables the design of highly precise and effective metalens that meets the desired phase requirements.

## 2.2 Deblurring CNN

The D-CNN module shown in Figure 2 utilizes the sensor output ($I^s$) from the optical layer as its input. Although many network architectures exist for this problem, we chose the well-known U-Net [24] as it is widely used in biomedical imaging for image reconstruction [13, 30, 32]. In short, the U-Net implementation has an encoder and decoder architecture with 23 convolution layers and 32 to 512 feature channels. At each step of the encoder stage, the input underwent two 3×3 convolution layers, a rectified linear unit (ReLU), and batch normalization (BN). Subsequently, the feature map is downscaled by a 2×2 max-pooling operation. Likewise, following two 3×3 convolution layers that incorporated ReLU and BN at each iteration
of the decoder, a 2×2 transposed convolution operator upsampled the feature map. At the final layer, a 1x1 convolution is used to map each 32-component feature vector to the desired number of classes, which is 1 in the current case. Moreover, a hyperbolic tangent (tanh) activation is utilized to map the output to the range of [-1,1]. The residual image is then incorporated by addition to the sensor output. The Clamp layer further processes the resulting image to constrain the data from 0 to 1.

For memory efficiency, U-Net takes input as 256×256 blurry pixel images corresponding to different depths and outputs the enhanced images at each iteration. It should be noted that to avoid border distortions during convolution with the PSF, the patched image and PSF are convolved and subsequently cropped to 256×256 after convolution. Once the network is trained, it can process images with dimensions that are multiples of 16 in width and height.

## 2.3 Implementation Details

The end-to-end framework is trained via an image dataset provided by Ref. 13. The dataset consists of a diverse array of microscopic fluorescent images of proflavin-stained oral cancer resections, histopathology images of healthy and cancerous tissues from the Cancer Genome Atlas (TCGA), and natural images from the National Digital Science and Technology Research Institute's (INRIA) Holiday dataset. This diverse selection ensures a broad range of feature scales within the dataset. The dataset contains a total of 1800 grayscale images, 1000×1000 pixels each, and includes 600 images of each type. The images are randomly assigned to training, validation, and testing datasets, with 1500, 150, and 150 images, respectively. During training, the images are randomly cropped and enhanced with rotation, flipping, and brightness adjustments for data augmentation. Throughout the training and validation stages, at each iteration, a random image patch is selected from the corresponding dataset and assigned to five uniformly separated depth positions within the diopter domain within the targeted scene depth boundaries.

The loss function used in the framework is the Root Mean Squared Error (RMSE) calculated at each pixel of the reconstructed-image stack in comparison to the corresponding pixel in the ground-truth image:

$$L_{RMSE} = \frac{1}{\sqrt{N}} \|I - I^R\|_2, \qquad (10)$$

where N is the number of pixels. As the input experiences blurring at varying defocus values throughout the depth, the framework intrinsically adapted the optics to achieve a defocus-insensitive PSF. Consequently, no explicit cost function is required to maintain PSF similarity within the designated depth range.

The PyTorch package is used to implement the framework, and stochastic gradient descent with the Adam optimizer [15] is employed for optimization. The learning rates for the Adam optimizer are selected empirically as 1e-7 for the optical layer and 1e-4 for the D-CNN. During end-to-end framework training, a two-step training process is used, where the initial step is training the U-Net with fixed optics. After the convergence, joint training of the optical rendering and D-CNN is performed to achieve optimal performance.

## 3 Results

Our design is aimed to mitigate the trade-off between DOF and spatial resolution for varying target DOFs by experimenting with different objective lenses within the 4f imaging setup, as illustrated in Figure 1. This trade-off is mathematically expressed as follows:

$$DOF \propto \frac{\lambda}{NA^2} \propto \frac{resolution^2}{\lambda}. \qquad (11)$$

This demonstrates that the spatial resolution is higher for a shallower DOF range; therefore, a shallower DOF range optimization problem is comparatively less challenging to solve. The complexity of the problem, however, is affiliated with the selected lenses and the desired DOF range. It is quantified and expressed
using the defocus coefficient ($\psi_z$) and presented alongside the system parameters for different simulations in Table 1.

<table><caption>Tab. 1: Test results and calculated system parameters for the selected lenses.</caption><thead><tr><th>Objective lens</th><th>NA</th><th>DOF</th><th>r (mm)</th><th>Δs(μm)</th><th>PSNR</th><th>SSIM</th><th>ψ<sub>max</sub></th></tr></thead><tbody><tr><td>RMS4X-PF*</td><td>0.13</td><td>± 100 μm</td><td>3.72</td><td>740</td><td>38.19</td><td>0.98</td><td>0.97</td></tr><tr><td>RMS4X-PF</td><td>0.13</td><td>± 1 mm</td><td>3.72</td><td>73</td><td>30.11</td><td>0.88</td><td>9.877</td></tr><tr><td>RMS20X-PF</td><td>0.40</td><td>± 100 μm</td><td>2.87</td><td>38</td><td>29.95</td><td>0.90</td><td>14.69</td></tr><tr><td>Mitutoyo50X</td><td>0.42</td><td>± 100 μm</td><td>1.07</td><td>20</td><td>30.84</td><td>0.88</td><td>10.48</td></tr><tr><td>Mitutoyo50X*</td><td>0.42</td><td>± 1 mm</td><td>1.07</td><td>1.54</td><td>29.28</td><td>0.89</td><td>136.24</td></tr><tr><td>Mitutoyo50X*</td><td>0.42</td><td>± 1.5 mm</td><td>1.07</td><td>0.858</td><td>27.98</td><td>0.89</td><td>245.54</td></tr></tbody></table>

Table 1 demonstrates that solving the EDOF problem for a higher NA lens system is considerably more challenging task, where the sampling requirements, determined by Eq. 8, become more demanding due to the broader bandwidth of the defocused pupil function, necessitating finer sampling for accurate results.

Such a case, thus, advocates the metalens to address the underlying stricter sampling requirement. Conversely, for the same EDOF range, using a low NA lens-based system with a learned DOE can yield a satisfactory solution. However, it should be noted that the final magnification would be significantly lower in such a system. Therefore, the lens selection should be based on the requirements of the sample to be visualized. To the best of our knowledge, certain biological specimens, such as the developing embryo, exhibit a diameter that increases from approximately 200 µm to nearly 3 mm [21]. As such, we have selected to target DOF ranges of 200 µm, 2 mm, and 3 mm for our study.

To further support the results, we compare the proposed algorithm with two existing methods. Initially, a cubic mask is adopted as the conventional method for DOF extension, which modulates the phase as

$$ \phi^M(x_1, y_1) = \text{mod} \left[ \frac{\alpha}{N^3} (x_1^3 + y_1^3), 2\pi \right], |x| < \frac{N}{2}, |y| < \frac{N}{2}, \qquad (12) $$

where $\alpha$ determines the number of $2\pi$ transitions. In this case, a fixed cubic phase mask is combined with a U-Net and subsequently trained to perform deblurring for all EDOF targets. To select an appropriate mask for each DOF range, the modulation transfer function (MTF) is evaluated across five different depths by tuning $\alpha$, to be insensitive to defocus and represent all spatial frequencies. $\alpha$ values of $6\pi$, $200\pi$, and $360\pi$ are selected for the DOF ranges of 200 µm, 2 mm, and 3 mm, respectively.

Additionally, the algorithm proposed by Jin et al. is adopted as a more recent advanced EDOF method based on the end-to-end learning framework and Zernike basis ($Z_n$) representation. Notably, the paper does not provide a relationship between the choice of the number of Zernike polynomials and the EDOF range. To align the optical setup parameters with our methodology and ascertain the appropriate number of parameters (n) for each depth range, we calculated the Fourier transform of the $Z_n$ and found the bandwidth of the resultant signal. Considering the Nyquist theorem, we calculated the sampling rate for the $Z_n$. Based on these calculations, we selected the first 11, 1100, and 2000 $Z_n$ for DOF targets of 200 µm, 2 mm, and 3 mm, respectively.

We present a quantitative analysis of the proposed method in comparison with existing approaches by analyzing the performance metrics, specifically peak signal-to-noise-ratios (PSNRs) and structural similarities (SSIMs). For each test setup, the PSNR and SSIM values are derived and depicted in Figure 4. Additionally, the mean PSNR and SSIM values are calculated for all images within the 150 test images of the dataset and presented in Table 2. To calculate the average values, we assign each test image to predetermined depths, uniformly distributed throughout the scene depth range. Subsequently, the PSNR and SSIM values are averaged. As demonstrated in Table 2, the proposed method outperforms existing approaches in terms of PSNR and SSIM values, both for the sample image shown in Figure 4 and the mean values across the entire test dataset. In particular, the D-CNN alone, as inferred from the fixed Cubic mask

<footnote>* Used as the optimized design for the targeted DOF range.</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 4:** Simulated performance of a cubic-mask-enabled computational 4f system, DeepDOF model, computational EDOF microscope, optimized for various target depths. For all DOFs, our method shows superior performance.</figcaption></figure>
and U-Net simulations, is inadequate. Upon increasing the target DOF, the image becomes excessively blurry at the sensor level, and the U-Net cannot recover the images effectively. Comparable results are visible in the DeepDOF approach, which is linked to the sub-sampling of the frequency domain that arises from using the Zernike basis representation. It is important to acknowledge that Ref. 13 is optimized using 55 Zernike basis, for a target DOF of 200 µm. The starred result in Table 2 displays the outcomes obtained when employing 55 Zernike basis. In this case, we retain the same setup parameters as in our other simulations, only modifying the number of Zernike basis utilized to represent the height map in the method, resulting in a higher sampling than necessary. The results demonstrate that increasing the sampling does not improve the performance, thus confirming that our calculated setup parameters sufficiently address the system complexity. Additionally, in our U-Net implementation, we adapted a modified U-Net architecture for the deblurring layer, diverging from the original approach. The main difference resides in the final layer of our U-Net architecture, where we incorporated the residual image and applied the clamp function, enhancing both the accuracy and efficiency of the deblurring process.

<table><caption>**Tab. 2:** Quantitative analysis of methods for EDOF problem. It should be noted that all the methods are simulated using the same system parameters that are computed for each DOF range. Refer to Table 1 for system parameters.</caption><thead><tr><th>Framework configuration</th><th>DOF</th><th>PSNR</th><th>SSIM</th></tr></thead><tbody><tr><td rowspan="3">Fixed cubic + UNet</td><td>± 100 µm</td><td>34.99</td><td>0.92</td></tr><tr><td>± 1 mm</td><td>23.6</td><td>0.62</td></tr><tr><td>± 1.5 mm</td><td>21.86</td><td>0.58</td></tr><tr><td rowspan="3">DeepDOF</td><td>± 100 µm</td><td>28.90, 28.06*</td><td>0.80, 0.79*</td></tr><tr><td>± 1 mm</td><td>20.67</td><td>0.55</td></tr><tr><td>± 1.5 mm</td><td>21.14</td><td>0.58</td></tr><tr><td rowspan="3">Our method</td><td>± 100 µm</td><td>38.19</td><td>0.98</td></tr><tr><td>± 1 mm</td><td>29.28</td><td>0.89</td></tr><tr><td>± 1.5 mm</td><td>27.98</td><td>0.89</td></tr></tbody></table>

An alternative method for evaluating EDOF performance involves examining the characteristics of MTFs across various depths, with MTFs representing the magnitudes of the Fourier transforms corresponding to their associated PSFs [17]. To facilitate efficient depth-agnostic deblurring throughout the whole depth range of interest, MTF pass-bands should be as wide as possible to recover features at various spatial frequencies while maintaining a high degree of similarity among themselves. Figure 5 presents the final metasurface designs for the 3 mm DOF scenario, as well as the MTFs at three distinct depths. The depth dependency of the MTFs in the proposed method is decreased compared to both the DeepDOF and Fixed cubic cases. The MTFs of the proposed method display greater similarity among themselves and avoid crossing zero, ensuring the preservation of information during recovery. This similarity is attributed to depth-agnostic deblurring assisting MTFs to remain consistent across all targeted depths. To compare the frequency support of each method, we defined a threshold of 0.1 and examine the MTF magnitude across the frequency spectrum and depths, which formally determines the maximum spatial resolution transmitted through the optics. The MTFs of the proposed method resulted in broadband coverage across the entire spectrum, illustrating that even at higher spatial frequencies, MTFs do not approach zero. This finding corresponds to the clear visibility of small structures. Conversely, at lower spatial frequencies, the MTF is closer to 1, representing the ability to clearly visualize large structures. Indeed, as demonstrated in the other methods, all MTFs can clearly visualize large structures, but for smaller structures, quality is inadequate.

<footnote>* Used first 55 Zernike polynomials.</footnote>
<figure><img src="image_7.png" /><figcaption>**Fig. 5:** The optimized metalens radius maps, as well as the MTFs at three distinct depths, are presented for the existing and proposed methods, encompassing the maximum defocus values within the 3 mm DOF scene and the in-focus depth.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>**Fig. 6:** Reconstruction results for increasing radius and height-map fabrication error levels.</figcaption></figure>
## 3.1 Fabrication Error Analysis

The robustness of the proposed EDOF microscope model is tested considering various levels of inaccuracies in the fabrication of metasurface and DOE. In particular, the effects of such fabrication inaccuracies are modelled by introducing random Gaussian-distributed noise with a standard deviation $\sigma_r$, to the optimized optical elements, as the radius and DOE height error. The tested metasurface radius deviation levels are $\sigma_r = 5nm, 12nm$, while the DOE height deviation is assumed as $\sigma_h = 30nm, 50nm$. Figure 6 presents the outcomes for the various EDOF configurations. As it can be inferred from the figure, the post-processing method performs adequately for lower fabrication error levels but exhibits limited robustness in the presence of higher error margins, leading to a noticeable reduction in image quality, particularly at the boundaries of the targeted EDOF. The robustness of the algorithm can be increased by incorporating fabrication error boundaries during training or retraining the D-CNN following the fabrication of optical components. Additionally, the implementation of advanced denoising algorithms can contribute to the enhancement of the results.

## 4 Future work and Conclusion

In conclusion, we implemented an end-to-end optimization framework that effectively addresses the physical limitations inherent in the 4f system EDOF microscope. Our approach offers two primary advantages over existing methods. Firstly, the incorporation of metasurface optics within our system has facilitated the achievement of the most extensive EDOF range reported in the literature thus far. In extreme EDOF scenarios, specifically when employing a higher NA lens, meta-optics offer a distinct advantage due to their increased design flexibility and superior SBP. Secondly, our method effectively minimizes defocus by systematically incorporating relevant optical system parameters throughout the optimization process. The results obtained from our simulations demonstrate that our proposed technique outperforms the current state-of-the-art methods in EDOF microscopy imaging, delivering consistently high performance across a broad range of EDOF values.

In future work, we aim to investigate the co-design of optics and post-processing for broadband EDOF imaging. Moreover, we aim to explore the application of our approach in addressing the challenges associated with light field microscopy. Our next objective is fabricating a selected metasurface, followed by a comprehensive evaluation of its performance within the context of the 4f system setup. We believe that these research directions will contribute significantly to the ongoing advancement of microscopy imaging, ultimately leading to more sophisticated and versatile optical systems.

**Acknowledgment:** This project has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 956770. This work was supported by the Academy of Finland (project no. 336357, PROFI 6 - TAU Imaging Research Platform).

**Funding:** Please insert information concerning research grant support here (institution and grant number). Please provide for each funder the funder's DOI according to https://doi.crossref.org/funderNames?mode=list.

## References

[1] S. Abrahamsson, J. Chen, B. Hajj, S. Stallinga, A. Y. Katsov, J. Wisniewski, G. Mizuguchi, P. Soule, F. Mueller, C. D. Darzacq, et al. Fast multicolor 3d imaging using aberration-corrected multifocus microscopy. *Nature methods*, 10 (1):60–63, 2013.

[2] U. Akpinar, E. Sahin, M. Meem, R. Menon, and A. P. Gotchev. Learning wavefront coding for extended depth of field imaging. *IEEE Transactions on Image Processing*, 30:3307–3320, 2019.
[3] A. Arbabi, Y. Horie, M. Bagheri, and A. Faraon. Dielectric metasurfaces for complete control of phase and polarization with subwavelength spatial resolution and high transmission. *Nature nanotechnology*, 10(11):937-943, 2015.

[4] E. Bayati, R. Pestourie, S. Colburn, Z. Lin, S. G. Johnson, and A. Majumdar. Inverse designed metalenses with extended depth of focus. *ACS photonics*, 7(4):873-878, 2020.

[5] S. Colburn, A. Zhan, and A. Majumdar. Metasurface optics for full-color computational imaging. *Science advances*, 4 (2):eaar2114, 2018.

[6] O. Cossairt and S. Nayar. Spectral focal sweep: Extended depth of field from chromatic aberrations. In *2010 IEEE International Conference on Computational Photography (ICCP)*, pages 1-8. IEEE, 2010.

[7] E. R. Dowski and W. T. Cathey. Extended depth of field through wave-front coding. *Applied optics*, 34(11):1859-1866, 1995.

[8] J. Engelberg and U. Levy. The advantages of metalenses over diffractive lenses. *Nature communications*, 11(1):1991, 2020.

[9] S. Geissbuehler, A. Sharipov, A. Godinat, N. L. Bocchio, P. A. Sandoz, A. Huss, N. Jensen, S. Jakobs, J. Enderlein, F. G. van der Goot, E. A. Dubikovskaya, T. Lasser, and M. Leutenegger. Live-cell multiplane three-dimensional super-resolution optical fluctuation imaging. *Nature Communications*, 5, 2014.

[10] A. K. Glaser, N. P. Reder, Y. Chen, E. McCarty, C. Yin, L. Wei, Y. W. Wang, L. D. True, and J. T. C. Liu. Light-sheet microscopy for slide-free non-destructive pathology of large clinical specimens. *Nature biomedical engineering*, 1, 2017.

[11] L. Huang, J. Whitehead, S. Colburn, and A. Majumdar. Design and analysis of extended depth of focus metalenses for achromatic computational imaging. *Photonics Research*, 8(10):1613-1623, 2020.

[12] M. Jang, Y. Horie, A. Shibukawa, J. Brake, Y. Liu, S. M. Kamali, A. Arbabi, H. Ruan, A. Faraon, and C. Yang. Wavefront shaping with disorder-engineered metasurfaces. *Nature photonics*, 12(2):84-90, 2018.

[13] L. Jin, Y. Tang, Y. Wu, J. B. Coole, M. T. Tan, X. K. Zhao, H. Badaoui, J. T. Robinson, M. D. Williams, A. Gillenwater, R. R. Richards-Kortum, and A. Veeraraghavan. Deep learning extended depth-of-field microscope for fast and slide-free histology. *Proceedings of the National Academy of Sciences of the United States of America*, 117:33051-33060, 2020.

[14] M. Khorasaninejad, A. Y. Zhu, C. Roques-Carmes, W. T. Chen, J. Oh, I. Mishra, R. C. Devlin, and F. Capasso. Polarization-insensitive metalenses at visible wavelengths. *Nano letters*, 16(11):7229-7234, 2016.

[15] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. *CoRR*, abs/1412.6980, 2014.

[16] V. N. Le, S. Chen, and Z. Fan. Optimized asymmetrical tangent phase mask to obtain defocus invariant modulation transfer function in incoherent imaging systems. *Optics letters*, 39(7):2171-2174, 2014.

[17] A. Levin, R. Fergus, F. Durand, and W. T. Freeman. Image and depth from a conventional camera with a coded aperture. *ACM transactions on graphics (TOG)*, 26(3):70-es, 2007.

[18] D. Lin, P. Fan, E. Hasman, and M. L. Brongersma. Dielectric gradient metasurface optical elements. *science*, 345 (6194):298-302, 2014.

[19] Z. Lin, C. Roques-Carmes, R. Pestourie, M. Soljačić, A. Majumdar, and S. G. Johnson. End-to-end nanophotonic inverse design for imaging and polarimetry. *Nanophotonics*, 10(3):1177-1187, 2020.

[20] S. Liu and H. Hua. Extended depth-of-field microscopic imaging with a variable focus microscope objective. *Opt. Express*, 19(1):353-362, Jan 2011. 10.1364/OE.19.000353. URL https://opg.optica.org/oe/abstract.cfm?URI=oe-19-1-353.

[21] J. Madhusoodanan. Smart microscopes spot fleeting biology. *Nature*, 614:378-380, 2023.

[22] J. N. Mait, R. A. Athale, J. van der Gracht, and G. W. Euliss. Potential applications of metamaterials to computational imaging. In *Frontiers in Optics*, pages FTu8B-1. Optical Society of America, 2020.

[23] Y. Peng, Q. Sun, X. Dun, G. Wetzstein, W. Heidrich, and F. Heide. Learned large field-of-view imaging with thin-plate optics. *ACM Trans. Graph.*, 38(6):219-1, 2019.

[24] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. *ArXiv*, abs/1505.04597, 2015.

[25] W. J. Shain, N. A. Vickers, B. B. Goldberg, T. G. Bifano, and J. Mertz. Extended depth-of-field microscopy with a high-speed deformable mirror. *Optics letters*, 42 5:995-998, 2017.

[26] S. S. Sherif, W. T. Cathey, and E. R. Dowski. Phase plate to extend the depth of field of incoherent hybrid imaging systems. *Applied optics*, 43(13):2709-2721, 2004.

[27] V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. P. Boyd, W. Heidrich, F. Heide, and G. Wetzstein. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. *ACM Transactions on Graphics (TOG)*, 37:1-13, 2018.

[28] T. Stone and N. George. Hybrid diffractive-refractive lenses and achromats. *Applied optics*, 27(14):2960-2971, 1988.

[29] H.-Y. Sung, S. S. Yang, and H. Chang. Design of mobile phone lens with extended depth of field based on point-spread function focus invariance. In *Novel Optical Systems Design and Optimization XI*, volume 7061, pages 65-75. SPIE, 2008.

[30] S. Tan, F. Yang, V. Boominathan, A. Veeraraghavan, and G. V. Naik. 3d imaging using extreme dispersion in optical metasurfaces. *ACS Photonics*, 8(5):1421-1429, 2021.
[31] E. Tseng, S. Colburn, J. Whitehead, L. Huang, S.-H. Baek, A. Majumdar, and F. Heide. Neural nano-optics for high-quality thin lens imaging. *Nature communications*, 12(1):6493, 2021.

[32] J. P. Vizcaíno, F. Saltarin, Y. Belyaev, R. Lyck, T. Lasser, and P. Favaro. Learning to reconstruct confocal microscopy stacks from single light field images. *IEEE transactions on computational imaging*, 7:775-788, 2021.

[33] Y. Wu, V. Boominathan, H. Chen, A. C. Sankaranarayanan, and A. Veeraraghavan. Phasecam3d — learning phase masks for passive single view depth estimation. *2019 IEEE International Conference on Computational Photography (ICCP)*, pages 1-12, 2019.

[34] S. Xiao, H. an Tseng, H. J. Gritton, X. Han, and J. Mertz. Video-rate volumetric neuronal imaging using 3d targeted illumination. *Scientific Reports*, 8, 2018.

[35] N. Yu and F. Capasso. Flat optics with designer metasurfaces. *Nature materials*, 13(2):139-150, 2014.

[36] A. Zhan, S. Colburn, C. M. Dodson, and A. Majumdar. Metasurface freeform nanophotonics. *Scientific reports*, 7(1): 1-9, 2017.

[37] H. Zhao and Y. Li. Optimized sinusoidal phase mask to extend the depth of field of an incoherent imaging system. *Optics letters*, 35(2):267-269, 2010.

[38] G. Zheng, R. Horstmeyer, and C. Yang. Wide-field, high-resolution fourier ptychographic microscopy. *Nature photonics*, 7:739-745, 2013.

[39] G. Zheng, H. Mühlenbernd, M. Kenney, G. Li, T. Zentgraf, and S. Zhang. Metasurface holograms reaching 80% efficiency. *Nature nanotechnology*, 10(4):308-312, 2015.