Article

# Embedded Processing for Extended Depth of Field Imaging Systems: From Infinite Impulse Response Wiener Filter to Learned Deconvolution

Alice Fontbonne<sup>1,*</sup> (D), Pauline Trouvé-Peloux<sup>2 ID</sup>, Frédéric Champagnat<sup>2 ID</sup>, Gabriel Jobert<sup>3</sup>
and Guillaume Druart<sup>1</sup>

**Abstract:** Many works in the state of the art are interested in the increase of the camera depth of field (DoF) via the joint optimization of an optical component (typically a phase mask) and a digital processing step with an infinite deconvolution support or a neural network. This can be used either to see sharp objects from a greater distance or to reduce manufacturing costs due to tolerance regarding the sensor position. Here, we study the case of an embedded processing with only one convolution with a finite kernel size. The finite impulse response (FIR) filter coefficients are learned or computed based on a Wiener filter paradigm. It involves an optical model typical of codesigned systems for DoF extension and a scene power spectral density, which is either learned or modeled. We compare different FIR filters and present a method for dimensioning their sizes prior to a joint optimization. We also show that, among the filters compared, the learning approach enables an easy adaptation to a database, but the other approaches are equally robust.

**Keywords:** deconvolution; Wiener filter; end-to-end design

<figure><img src="image_1.png" /></figure>

check for
updates

**Citation:** Fontbonne, A.;

Trouvé-Peloux, P.; Champagnat, F.;
Jobert, G.; Druart, G. Embedded
Processing for Extended Depth of
Field Imaging Systems: From Infinite
Impulse Response Wiener Filter to
Learned Deconvolution. *Sensors* **2023**,
23, 9462. https://doi.org/10.3390/
s23239462

Academic Editors: Wei Li and
Honglu Jiang

Received: 10 October 2023

Revised: 22 November 2023

Accepted: 26 November 2023

Published: 28 November 2023

<figure><img src="image_2.png" /></figure>

**Copyright:** © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).

# 1. Introduction

Optical systems naturally have a limited depth of field (DoF), and it is a common problem to wish to increase it. This can be used either to see sharp objects from a greater distance or to reduce manufacturing costs due to tolerance regarding the sensor position. One way of doing this is to use a joint optical/digital design technique (or “co-design”). In this context, the variation of the optical system’s point spread function (PSF) as a function of the depth of field is reduced to the detriment of the image quality of the sensor. The use of a cubic phase mask is a typical way of achieving this PSF invariance [1]. Digital processing is then required for restoration [2] and (preferably) optimized in conjunction with the optical system. This means that digital processing parameters depend on optical system parameters, but it also means that optical system parameters depend on digital processing. Among the different digital processing techniques, neural networks can be used [3-9]. However, some applications (e.g., advanced driver assistance systems) require real-time digital processing, with a very limited memory storage ability. These applications are also very limited in terms of power consumption: certain industrial constraints limit camera power consumption (including digital processing) to 1 W. In this case, even approaches such as the use of an infinite impulse response (IIR), Wiener filter, or mixed deep learning/Wiener approaches [10-12] are no longer usable. For these applications, it is only possible to implement a linear filter with a finite impulse response (FIR) [13-15]. Indeed, current industrial constraints for low-consumption, embedded real-time systems require the use of FPGA boards [16,18]? . Storage (for example, for recording a deconvolution kernel) is limited, and operations have to be performed as the image is acquired (at almost every

<footnote n="1">1 DOTA, ONERA, Université Paris Saclay, 91123 Palaiseau, France</footnote>

<footnote n="2">2 DTIS, ONERA, Université Paris Saclay, 91123 Palaiseau, France</footnote>

<footnote n="3">3 LYNRED, Route de Valence, 38113 Veurey-Voroize, France</footnote>

<footnote>* Correspondence: alice fontbonne@onera.fr</footnote>
line); otherwise, the entire image would have to be stored. This drastically limits the weight of the restoration algorithm that can be used. Yet, in an end-to-end, optical/digital design context for DoF extension, the PSFs of the optical system are often large [1,19,20], which can be problematic if one wishes to deconvolve with a FIR filter, especially with a very small kernel.

In this article, we focus on a step preceding the co-optimization of a hybrid optical/digital system (Figure 1). One of these first steps is to dimension the real-time digital processing to be applied and co-optimized with the optical part (i.e., choose its preliminary specifications) in order to enhance the DoF [21]. The size of the deconvolution kernel is one of the variables to be chosen before codesign, in the same way as the number of lenses, the number of active surfaces, and the surface parameterization (aspherization, freeform, etc.) of an optical system are chosen before optimization begins. The aim of this article is twofold. The first is to propose a methodology for choosing the size of the deconvolution kernel. The second is to evaluate the potential impact of this choice on the result after the co-optimization of the imaging system. Our contributions are the following. We build a simple performance model that predicts the mean square error (MSE) after deconvolution. Based on this performance model, we quantitatively compare the improvement, in equivalent settings, provided by different FIR filter types: either a truncation of the IIR Wiener Filter, a FIR Wiener filter, or a FIR filter with learned coefficients. Furthermore, we evaluate the benefit of the learning approach compared to a model-based approach. In this article, we first describe in Section 2 the pipeline of the imaging system consisting of an optical system and different possible deconvolution filters. Then, in Section 3, we study the effect of different kernel sizes as a function of the PSF width and noise. This provides a method for determining an appropriate deconvolution kernel size. At the end of this results section, we focus on the contribution of learning the FIR filter coefficients for deconvolution on real images. Finally, we conclude on the evolution of the image quality as a function of the deconvolution kernel size for the different studied FIR filters.

<figure><img src="image_3.png" /><figcaption>**Figure 1.** A classical codesign loop iterates on optical system and filter parameters in order to optimize a merit function. The initial state is provided by the dimensioning task. The red box corresponds to the part studied in this article.</figcaption></figure>
## 2. Description of the Processing Pipeline

In this article, we consider a usual image restoration model $I(r) = w \star (h \star O + n)(r)$, where r values are the spatial (2D) coordinates, w(r) is the deconvolution filter, h(r) is the PSF, and * stands for the convolution operation. O(r) is the perfect scene image, which is assumed to be a zero-mean and unitary variance, as well as a stationary random process of power spectral density (PSD) $S_{OO}(v)$ (with v being the spatial frequency coordinates, which are normalized to the Nyquist frequency of the sensor). The noise n(r) is modelled as a zero-mean Gaussian distribution. Its standard deviation depends on the chosen signal to noise ratio (SNR). Its (constant) PSD is denoted as $S_{nn}(v)$. We measure the SNR in dB according to $SNR = 10 \log_{10} \frac{\int S_{OO}(v)dv}{\int S_{nn}(v)dv}$.

### 2.1. PSD Model and Image Database

Estimating the $O(r)$ from $I(r)$ is an ill-posed problem that requires the use of a priori knowledge about the $O(r)$, either via a model or a database. Both must be representative of the observed scene. We consider the following “Reichenbach PSD model” from [22]:

$$S_{OO}(v) = \frac{2\pi\mu^2}{(1 + 4\pi^2\mu^2v^2)^{3/2}}, \qquad (1)$$

with $1/\mu$ being homogeneous to a cutoff frequency and $\int S_{OO}(v)dv$ being normalized to one. $\mu$ is related to the mean spatial detail. The value of $\mu$ has been set to 15 in order to be in rough agreement with the PSD of the real scenes database described hereafter. In this paper, we consider two databases: one generated from a Reichenbach model described in Section 3 (Sections 3.1–3.3) and the other from an automotive database of real long-wave infrared images [23] described in Section 3.4. The first is directly generated using the definition of the power spectral density, which is multiplied in the Fourier domain to become a matrix of randomly drawn numbers. This results in images that are not visually representative of real scenes (Figure 2) but have a controlled, realistic spectral content. The second is composed of 5943 images of size 640 × 480. It has been captured while driving in an urban environment (with pedestrians and cars at a distance of about 1 to 30 m) both during the day and at night. Examples of the images from this database are given in Figure 3.

<figure><img src="image_4.png" /><figcaption>Figure 2. (a) An image generated from the Reichenbach PSD model. (b) Another image generated from the same model.</figcaption></figure>
<figure><img src="image_5.png" /><figcaption>Figure 3. First row: two random full images from the database. Second row: four random patches from the database.</figcaption></figure>

## 2.2. PSF Model for System with Extended DoF

In this article, we aim to dimension a system that will be codesigned as described in Figure 1. In particular, this dimensioning concerns the choice of the deconvolution filter size and shape, which depend a priori on the size of the kernel to be deconvolved (the PSF). However, the PSF is unknown, as the optical system is still at the dimensioning stage. Therefore, a model is needed to predict the general shape of the PSF and its spread over the sensor. Here, the model is established from previous codesign results based on the use of phase masks in the pupil of a system at the diffraction limit, such as cubic phase masks, annular binary phase masks, or other shapes [24]. These systems, with their extended DoF values, have the particularity of presenting very specific modulation transfer functions (MTFs) with a sharp drop-off at low spatial frequencies and signal conservation at medium and high spatial frequencies [25].

We propose to account for this behavior with the following model for all the MTFs through their DoF values:

$$ \tilde{h}_{\rho,a}(v) = a \times (1 - |v|)^{\rho} + (1 - a) , \qquad (2) $$

with $\tilde{\cdot}$ denoting the Fourier transform and $a$ and $\rho$ denoting the two parameters of the model. $(1 - a)$ is the value of the MTFs at the normalized Nyquist frequency. $\rho$ is a parameter related to the expected DoF extension: $\tilde{h}_{1,a}(v)$ models a kind of diffraction-limited optical system (with a low DoF) as it becomes an affine function. An example is given in Figure 4a. The larger is $\rho$, the faster the drop in the MTF. It enables us to reach an enhanced DoF. Figure 4b shows the example of a system with a cubic phase mask [1], thereby aiming to greatly extend the DoF.

To give another example of the model's reliability, Figure 5 shows the MTFs of an extended DoF-F/0.8 system (with a phase mask derived from [2]), which is invariant with respect to the field of view and DoF. Note that this system on an axis and without a phase mask is diffraction-limited, given an MTF at 0.4 in the Nyquist frequency. In this latter case and with a binary phase mask, we also observe a sharp drop for the low frequencies and a preservation of the high frequencies (with no nullings), which are characteristics of codesigned systems. This drop in the MTFs for low frequencies naturally generates a wider base for the PSF compared to a diffraction-limited optical system. Therefore, the PSF energy is less concentrated around the origin for a codesigned system (Figures 4 and 5).
<figure><img src="image_6.png" /><figcaption>**Figure 4.** (a) $\tilde{h}_{1,0,9}(v)$ with regard to the MTF of a diffraction-limited system (with low DoF) and the corresponding PSF with a $5 \times 5$ support size. (b) $\tilde{h}_{12,0,9}(v)$ with regard to the MTFs of an extended DoF system with a cubic phase mask. "Before focus" and "After focus" curves are superimposed.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 5.** $\tilde{h}_{6,0,9}(v)$ with regard to the MTF of an extended DoF-F/0.8 system (with phase mask derived from [2]) and the corresponding PSF with large base centered on a $5 \times 5$ support size. "Before focus" and "After focus" are in the DoF extension optimized range of the phase mask.</figcaption></figure>

In the rest of the article, we set the parameter *a* to 0.9, thus considering that if the MTF were to take values below 0.1, it would be below the noise levels for most realistic applications [25].

## 2.3. The MSE Criterion

In this article and otherwise stated, the MSE (between the ideal image of the scene and the final image [25]) is the optimization and evaluation criterion before and after processing. Any evaluation of "the best filter" is therefore made in the sense of the MSE. It is computed
by its analytical formula in Section 3 and empirically for the images at the end of this results section.

## 2.4. Different FIR Filters

To restore an image with a linear deconvolution filter, the best filter is the IIR Wiener filter (which requires the a priori knowledge of $S_{OO}(v)$ for its construction):

$$ \tilde{w}(v) = \frac{\tilde{h}^*(v)}{|\tilde{h}(v)|^2 + \frac{s_{nn}(v)}{s_{OO}(v)}} \quad (3) $$

with $\tilde{h}^*(v)$ being the complex conjugate of $\tilde{h}(v)$. It gives the best achievable MSE with a linear filter, but this filter has an infinite support, which makes it unusable with the hardware constraints considered in this article. In addition to this filter, we are interested in three types of FIR filters, which respect the hardware constraints. The three filters are built knowing the size $K \times K$ of the filter ($K$ is odd), the model of the PSF, and the SNR:

* The simplest approach is to truncate the IIR Wiener filter, in the direct space, to the chosen kernel size $K \times K$. It will be noted as the “Truncated IIR Wiener filter” in the following. Since we consider small kernel sizes, we use a simple rectangular windowing and no apodization, which is in contrast with [26–29].

* A second approach consists of optimizing a “FIR Wiener filter”, i.e., finding the best linear filter under the constraint of a finite kernel size [22]. It needs the autocorrelation of the scene for its construction, which is calculated from the scene PSD using a Fourier transform.

* Finally, the “learned filter” corresponds to the minimization over the $K \times K$ filter coefficients of the MSE criterion averaged on an image database (which plays the role of the scene PSD). We use the Adam optimizer for that, with a batch size of 10 and 0.01 as the learning rate. The choice of starting point will be discussed in Section 3.

# 3. Results

In this section, we compare the deconvolution performance outcomes of different FIR filters with respect to various cases. The scene was first represented by the Reichenbach model, from which we generated a training dataset for learning.

## 3.1. First Case Study

We compared the three types of filters presented in Section 2 in a first case where the deconvolution kernel remained large ($17 \times 17$) and with $\rho = 8$ and SNR = 34 dB. We first compared the different effective MTFs (defined by $|\tilde{h}(v) \times \tilde{w}(v)|$), i.e., the transfer functions after deconvolution, which are shown in Figure 6. Both the Truncated IIR Wiener filter and the FIR Wiener filter were close to the IIR Wiener filter, but the first one clearly modified the mean of the image, while the second oscillated around the best effective MTF. We tested several starting points for the learning approach: either starting from random coefficients, from the Truncated IIR filter, or directly from the FIR filter. The learned filter, when starting from the FIR Wiener filter, showed no change in the effective MTF. Indeed, the FIR Wiener filter already corresponded to the theoretical optimal filter. Starting the learning from the random or Truncated IIR Wiener filter converged to solutions that were close to the FIR Wiener filter, in particular for the lowest frequencies, but that were differ slightly for the highest ones. The optimization was also much longer when the starting point was random. Reaching the theoretical optimal solution using learning requires a careful choice of the starting point and of the learning hyperparameters (e.g., by increasing the batch size and using another optimizer specifically for this problem, such as via LBFGS).

In terms of the MSE (Table 1), the filters learned from the FIR Wiener filter and from the Truncated IIR Wiener filter yielded equivalent results that were close to the optimal one (the IIR Wiener filter). Hence, in the following, we systematically considered the Truncated
IIR Wiener filter as the starting point for learning: this filter is close to the IIR Wiener filter, except for at very low spatial frequencies (Figure 6).

<figure><img src="image_8.png" /><figcaption>**Figure 6.** Effective MTFs calculated from the different IIR and FIR filters limited to a 17 × 17 kernel.</figcaption></figure>

<table><caption>**Table 1.** First row: MSEs of the different filters. Second row: MSEs of the filters learned from first row.</caption><thead><tr><th></th><th>IIR Wiener</th><th>Truncated IIR Wiener</th><th>FIR Wiener</th><th>Random</th></tr></thead><tbody><tr><td>Before learning</td><td>0.0146</td><td>0.0210</td><td>0.0150</td><td>-</td></tr><tr><td>After learning</td><td>-</td><td>0.0150</td><td>0.0150</td><td>0.0160</td></tr></tbody></table>

## 3.2. Impact of Deconvolution Kernel Size on Image Quality

We expanded the previous study to different kernel sizes while still having $\rho = 8$ and $SNR = 34$ dB. Figure 7 shows the variation in the MSE with respect to the kernel width $K$ for the three methods. The MSE globally decreased when $K$ increased. This is an expected result for the FIR filter [22], and Figure 7 confirms that, even though the learned filter had a different shape (Figure 6), it yielded similar results to the best (FIR) filter. The Truncated IIR Wiener filter yielded the worse MSE performance, especially for very small kernels. We noticed an asymptotic behavior, with no need to use a large kernel to have a good reconstruction. For this case study, $K = 5$ was sufficient to achieve 95% of the best result with a linear filter, which was obtained with the IIR Wiener filter. This 95% threshold is plotted as shown by the dotted line in Figure 7.

## 3.3. Method for Choosing the Kernel Size

We reproduced this study for different $\rho$ values, different SNR values, and different kernel sizes. With a similar 95% threshold as in Figure 7, we could deduce the minimal size of the deconvolution kernel to obtain nearly the best image quality with a linear filter (Figure 8). This minimal size was obtained with the FIR Wiener filter, as well as with the learned filter. With respect to increasing the DoF (with increasing $\rho$), the more abruptly the MTF dropped for low frequencies, the larger the base of the PSF was, and the larger the kernel of the deconvolution filter needed to be. Furthermore, a larger $K$ was needed for low SNR values. Indeed, when the noise is very important, the reconstruction is done mainly
for the low spatial frequencies. The deconvolution filter (in the Fourier space) is therefore tightened even if the PSF is large. With such an approach, a deconvolution filter of size 5 × 5 appears sufficient when the thermal contrast is good (34 dB), even in the context of increasing the DoF by codesign. It will however be necessary to increase this size for lower SNR values. For example, a size of 9 × 9 is preferable at 20 dB.

<figure><img src="image_9.png" /><figcaption>**Figure 7.** Theoretical evolution of the MSE with different filters as a function of the deconvolution kernel size in the case of a Reichenbach PSD, with $\rho = 8$ and $SNR = 34$ dB.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>**Figure 8.** (a) Size of the kernel depending on $\rho$ (b) Size of the kernel depending on $SNR$ to reach the 95% threshold.</figcaption></figure>

### 3.4. Robustness on Real Images

We now consider the case where the true scene statistics differs substantially from the Reichenbach model. Learning makes it possible to take the differences between a real scene and an analytic model into account. We consider here an automotive database. Over the entire database, we can calculate an average MSE (MMSE) that is equivalent to the theoretical MSE calculated above. As shown in Figure 9, we obtained an evolution similar to the one studied in Section 3; here, this was obtained with $\rho = 8$ and $SNR = 20$ dB, while still considering the normalized Reichenbach PSD for Wiener filters. The Truncated IIR Wiener filter needed a large kernel size to be efficient, while the FIR Wiener filter quickly yielded asymptotic results. Learning this allowed us to improve the performance outcomes.
The associated MMSE curve followed that of the Wiener FIR filter but at a lower level: the learning process allowed us to adapt the filtering to the new PSD. Indeed, the FIR Wiener filter was no longer optimal, because the model used to define the filter was less representative of the database. Nevertheless, it remained relatively robust. The results can be seen in Figure 10. In this figure, we are interested in a subsection of an image from the database used. The various reconstructions were compared with the ground truth (or “ideal image”, which is shown in Figure 10a) and the simulation of the passage of this image through the optical system (taking noise into account, which is shown in Figure 10b). Qualitatively, the reconstruction with the FIR filter seems to give the most details. However, the noise was also amplified and yielded an orange peel effect (Figure 10e). This effect is visible regardless of the size of the deconvolution filter (Figure 10e–h). The learned filter made a better compromise in the sense of the MSE (Figure 10d): the image is less sharp, but the noise is less amplified. Note that similar results can be observed with the other images in the database and also with other databases [30].

<figure><img src="image_11.png" /><figcaption>**Figure 9.** Variation in the MMSE for the test dataset with respect to the deconvolution kernel size with respect to the automotive database.</figcaption></figure>

<figure><img src="image_12.png" /><figcaption>**Figure 10.** Patch of an image of the database, with $\rho = 8$ and SNR = 20 dB. (a) Ideal image. (b) Simulated acquired image. Deconvolution with a 9 × 9 kernel size. (c) Truncated Wiener filter. (d) Learned filter. (e) FIR Wiener filter. (f–h): other FIR filter sizes.</figcaption></figure>
For the case of the 9 × 9 kernel, the Truncated Wiener filter yielded good visual results despite a strong MSE. This was not the case when the kernel was very small (<7 × 7), but, when the kernel was 9 × 9 in size (or larger, see Figure 6), this result was expected. Indeed, according to the effective MTFs shown in Figure 6, mainly the very low frequencies were not well reconstructed: the mean value of the reconstructed image is wrong, but this has little visual impact. We quantified the impact of processing on the database using two typical and complementary image processing metrics: the MMSE and the structural similarity index measure (SSIM) [31]. As shown in Table 2, this bad reconstruction of the mean had an impact on both the MSE and the SSIM, but this impact was lower for the second metric.

<table><caption>**Table 2.** Average values of MMSE and SSIM for patches of automotive database for deconvolution with different 9 × 9 filters, with $\rho = 8$ and $SNR = 20$ dB.</caption><thead><tr><th></th><th>No Filter</th><th>Truncated IIR Wiener</th><th>FIR Wiener</th><th>Learned Filter</th></tr></thead><tbody><tr><td>MMSE</td><td>0.0642</td><td>0.1591</td><td>0.0234</td><td>0.0177</td></tr><tr><td>SSIM</td><td>0.3984</td><td>0.4825</td><td>0.5253</td><td>0.6039</td></tr></tbody></table>

Therefore, this learning approach allowed for an improvement in the MSE by better taking into account the true PSD of the scene, whereas the Wiener approaches used a template PSD. Even if the MSE was the optimization loss, the obtained filter was also better on the SSIM. The Wiener FIR filter still offered good visual performance outcomes, even for very small deconvolution kernel sizes.

## 4. Conclusions

In this article, we have used FIR filters with codesigned systems. In particular, we have compared FIR filters for the image restoration of an optical system with an extended DoF (i.e., with a large base PSF). For the learning approach, we have shown the interest of initializing the learning with a truncated Wiener filter, both from the point of view of the learning time and the result found. Furthermore, we have established a method to determine a minimum kernel size as a function of a model of the noise and the MTFs of an extended DoF system based on the Wiener FIR filter results. We found that, depending on the noise level, a filter size of 5 × 5 to 9 × 9 could be sufficient for most DoF extensions using a Wiener FIR filter or a learned filter. This is a very important step in the design of a hybrid optical digital system. Finally, we have seen that the learning process can further improve the results through small adaptations to the database characteristics, but these improvements remain small. Indeed, this improvement is mainly for the high spatial frequencies whose restoration depends on the noise level.

The perspectives of this work are numerous. First, the actual embedding of the deconvolution filter requires the quantification of the coefficients of the deconvolution kernel, and this involves a study of the impact of this quantification on the image quality. This quantification could further limit the MSE, and this is the main limitation to this work. The second, and most important one, is to achieve the end-to-end optimization of an imaging system (optics and processing) starting from a well-chosen starting point: it is now possible to choose a kernel size that is well adapted to each DoF extension problem. It could also allow us to use metrics that are closer to visual perception. In addition, it will be possible to consider a live adaptability with several operating modes depending on the SNR.

**Supplementary Materials:** The following supporting information can be downloaded at: https://www.mdpi.com/article/10.3390/s23239462/s1: Figure S1: PSD as a function of the (normalized) spatial frequency of two scenes of the Infrared database and modeled by the Reichenbach model with $\mu = 15$.; Figure S2: (a,b) Sample 2D PSFs corresponding of those of Figure 4 of the article (with a logarithmic scale).; Figure S3: Modulus of filters used in Figure 6. (a) IIR Wiener Filter (b) Truncated IIR Wiener filter (c) FIR Wiener filter (d) Learned filter (from random) (e) Learned filter
(from Truncated IIR Wiener filter) (f) Learned filter (from FIR Wiener filter); Figure S4: Cross-section of the different IIR and FIR filters, limited to a 17 × 17 kernel.; Figure S5: Filters used in Section 3, for deconvolution of images (Figure 10). (a) Truncated IIR Wiener filter (b) FIR Wiener filter (c) Learned filter; Figure S6: First row: Ideal image (patch) or “ground truth”. Second row: Simulation through the optical system, with 20 dB SNR and deconvolution with the 9 × 9 FIR filter. (a,b) A random patch of the database; (c,d) Another random patch; (e,f) Another random patch; (g,h) Another random patch.

**Author Contributions:** Conceptualization, A.F. and P.T.-P.; methodology, A.F.; software, A.F., P.T.-P. and F.C.; validation, A.F.; formal analysis, A.F.; investigation, A.F.; resources, G.J.; data curation, G.J.; writing—original draft preparation, A.F.; writing—review and editing, A.F., P.T.-P., F.C., G.J. and G.D.; visualization, A.F.; supervision, A.F.; project administration, A.F. and G.D.; funding acquisition, G.D. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The code underlying the results presented in this paper is not publicly available at this time but may be obtained from the authors upon reasonable request. The automotive dataset is not publicly available.

**Acknowledgments:** We deeply thank the reviewers for their comments, which helped improve this article. Interested readers can find additional figures in the Supplementary Materials section.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Abbreviations**

The following abbreviations are used in this manuscript:

DoF Depth of Field

PSF Point Spread Function

IIR Infinite Impulse Response

FIR Finite Impulse Response

SNR Signal-to-Noise Ratio

MTF Modulation Transfer Function

MSE Mean Square Error

PSD Power Spectral Density

**References**

1. Dowski, E.R.; Cathey, W.T. Extended depth of field through wave-front coding. *Appl. Opt.* **1995**, *34*, 1859. [CrossRef] [PubMed]

2. Fontbonne, A.; Sauer, H.; Goudail, F. End-to-end optimization of optical systems with extended depth of field under wide spectrum illumination. *Appl. Opt.* **2022**, *61*, 5358. [CrossRef] [PubMed]

3. Elmalem, S.; Giryes, R.; Marom, E. Learned phase coded aperture for the benefit of depth of field extension. *Opt. Express* **2018**, *26*, 15316. [CrossRef] [PubMed]

4. Sun, Q.; Wang, C.; Fu, Q.; Dun, X.; Heidrich, W. End-to-end complex lens design with differentiate ray tracing. *ACM Trans. Graph.* **2021**, *40*, 1–13. [CrossRef]

5. Tseng, E.; Mosleh, A.; Mannan, F.; St-Arnaud, K.; Sharma, A.; Peng, Y.; Braun, A.; Nowrouzezahrai, D.; Lalonde, J.F.; Heide, F. Differentiable Compound Optics and Processing Pipeline Optimization for End-to-end Camera Design. *ACM Trans. Graph.* **2021**, *40*, 1–19. [CrossRef]

6. Halé, A.; Trouvé-Peloux, P.; Volatier, J.B. End-to-end sensor and neural network design using differential ray tracing. *Opt. Express* **2021**, *29*, 34748. [CrossRef]

7. Li, Y.; Wang, J.; Zhang, X.; Hu, K.; Ye, L.; Gao, M.; Cao, Y.; Xu, M. Extended depth-of-field infrared imaging with deeply learned wavefront coding. *Opt. Express* **2022**, *30*, 40018. [CrossRef]

8. Zhang, R.; Tan, F.; Hou, Q.; Li, Z.; Sun, Z.; Yang, C.; Gao, X. End-to-end learned single lens design using improved Wiener deconvolution. *Opt. Lett.* **2023**, *48*, 522. [CrossRef]

9. Yang, X.; Fu, Q.; Heidrich, W. Curriculum Learning for ab initio Deep Learned Refractive Optics. *arXiv* **2023**, arXiv:2302.01089. Available online: http://xxx.lanl.gov/abs/2302.01089 (accessed on 13 September 2023).

10. Dong, L.; Du, H.; Liu, M.; Zhao, Y.; Li, X.; Feng, S.; Liu, X.; Hui, M.; Kong, L.; Hao, Q. Extended-depth-of-field object detection with wavefront coding imaging system. *Pattern Recognit. Lett.* **2019**, *125*, 597–603. [CrossRef]
11. Makarkin, M.; Bratashov, D. State-of-the-Art Approaches for Image Deconvolution Problems, including Modern Deep Learning Architectures. *Micromachines* **2021**, *12*, 1558. [CrossRef] [PubMed]

12. Zhang, K.; Ren, W.; Luo, W.; Lai, W.S.; Stenger, B.; Yang, M.H.; Li, H. Deep Image Deblurring: A Survey. *Int. J. Comput. Vis.* **2022**, *130*, 2103–2130. [CrossRef]

13. Robinson, M.D.; Stork, D.G. Joint design of lens systems and digital image processing. In Proceedings of the International Optical Design Conference 2006, Vancouver, BC, Canada, 4–8 June 2006; Gregory, G.G., Howard, J.M., Koshel, R.J., Eds.; International Society for Optics and Photonics, SPIE: Bellingham, WA, USA, 2006; Volume 6342, p. 63421G. [CrossRef]

14. Bryant, K.; Edwards, W.D.; Rogers, R.K. Low-cost computational imaging infrared sensor. In Proceedings of the Infrared Imaging Systems: Design, Analysis, Modeling, and Testing XXV, Baltimore, MD, USA, 6–8 May 2014; Holst, G.C., Krapels, K.A., Ballard, G.H., Buford, J.A., Jr., Lee Murrer, R., Jr., Eds.; International Society for Optics and Photonics, SPIE: Bellingham, WA, USA, 2014; Volume 9071, p. 90710H. [CrossRef]

15. Burcklen, M.A.; Diaz, F.; Lepretre, F.; Rollin, J.; Delboulbe, A.; Lee, M.S.L.; Loiseaux, B.; Koudoli, A.; Denel, S.; Millet, P.; et al. Experimental demonstration of extended depth-of-field f/1.2 visible High Definition camera with jointly optimized phase mask and real-time digital processing. *J. Eur. Opt. Soc. Rapid Publ.* **2015**, *10*, 15046. [CrossRef]

16. Chou, C.J.; Mohanakrishnan, S.; Evans, J.B. FPGA implementation of digital filters. *Proc. Icspat. Citeseer* **1993**, *93*, 1.

17. Nayak, S.; Nayak, M.; Matri, S.; Sharma, K.P. Synthesis and Analysis of Digital IIR Filters for Denoising ECG Signal on FPGA. In *Evolving Networking Technologies*; John Wiley & Sons, Ltd.: Hoboken, NJ, USA, 2023; Chapter 12, pp. 189–210. [CrossRef]

18. Rajalakshmi, R.; Vishnupriya, G.; Sudharsanan, R.; Navaneethan, S.; Vijayakumari, P.; Karthikeyan, M.V. Digital Filter Design on High speed Communication with Low Power Criteria. In Proceedings of the 2023 International Conference on Computer Communication and Informatics (ICCCI), Coimbatore, India, 23–25 January 2023; pp. 1–5. [CrossRef]

19. Pinilla, S.; Miri Rostami, S.R.; Shevkunov, I.; Katkovnik, V.; Egiazarian, K. Hybrid diffractive optics design via hardware-in-the-loop methodology for achromatic extended-depth-of-field imaging. *Opt. Express* **2022**, *30*, 32633. [CrossRef] [PubMed]

20. Pant, L.; Singh, M.; Chandra, K.; Mishra, V.; Pandey, N.; Pant, K.; Khan, G.S.; Sakher, C. Development of cubic freeform optical surface for wavefront coding application for extended depth of field Infrared camera. *Infrared Phys. Technol.* **2022**, *127*, 104377. [CrossRef]

21. Lopez-Ramirez, M.; Ledesma-Carrillo, L.; Cabal-Yepez, E.; Botella, G.; Rodriguez-Donate, C.; Ledesma, S. FPGA-based methodology for depth-of-field extension in a single image. *Digit. Signal Process.* **2017**, *70*, 14–23. [CrossRef]

22. Reichenbach, S.; Park, S. Small convolution kernels for high-fidelity image restoration. *IEEE Trans. Signal Process.* **1991**, *39*, 2263–2274. [CrossRef]

23. Balit, E.; Chadli, A. GMFNet: Gated Multimodal Fusion Network for Visible-Thermal Semantic Segmentation. 2020; pp. 1–4. Available online: https://neovision.fr/wp-content/uploads/2021/02/Papier-ECCV.pdf (accessed on 7 February 2023).

24. Sheng, J.; Cai, H.; Wang, Y.; Chen, X.; Xu, Y. Improved Exponential Phase Mask for Generating Defocus Invariance of Wavefront Coding Systems. *Appl. Sci.* **2022**, *12*, 5290. [CrossRef]

25. Fontbonne, A.; Sauer, H.; Goudail, F. Comparison of methods for end-to-end co-optimization of optical systems and image processing with commercial lens design software. *Opt. Express* **2022**, *30*, 13556. [CrossRef]

26. Harris, F. On the use of windows for harmonic analysis with the discrete Fourier transform. *Proc. IEEE* **1978**, *66*, 51–83. [CrossRef]

27. Saramaki, T. Finite Impulse Response Filter Design. In *Handbook for Digital Signal Processing*; Wiley-Interscience: Hoboken, NJ, USA, 1993.

28. Vollmerhausen, R. Design of finite impulse response deconvolution filters. *Appl. Opt.* **2010**, *49*, 5814. [CrossRef] [PubMed]

29. Uzo, H.N.; Nonyelu, H.U.; Eneh, J.N.; Ozue, T.I.; Anoliefo, E.C.; Chijindu, V.C.; Oparaku, O.U. FIR Filter Design using Raised Semi-ellipse Window Function. *Indones. J. Electr. Eng. Inform.* **2022**, *10*, 592–603. [CrossRef]

30. Cimpoi, M.; Maji, S.; Kokkinos, I.; Mohamed, S.; Vedaldi, A. Describing Textures in the Wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 23–28 June 2014.

31. Wang, Z.; Bovik, A.; Sheikh, H.; Simoncelli, E. Image Quality Assessment: From Error Visibility to Structural Similarity. *IEEE Trans. Image Process.* **2004**, *13*, 600–612. [CrossRef]

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.