# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Raphael Abele, Redouane El Moubtahij, Daniele Fronte, Pierre-Yvan Liardet, Jean-Luc Damoiseaux, Jean-Marc Boi, Djamal Merad

► To cite this version:

Raphael Abele, Redouane El Moubtahij, Daniele Fronte, Pierre-Yvan Liardet, Jean-Luc Damoiseaux, et al.. FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy. IEEE Photonics Journal, 2019, 11 (5), pp.1-17. 10.1109/JPHOT.2019.2940689. hal-02315710

HAL Id: hal-02315710

https://amu.hal.science/hal-02315710

Submitted on 23 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Raphael Abele, Redouane El Moubtahij, Daniele Fronte, Pierre-Yvan Liardet, Jean-Luc Damoiseaux, Jean-Marc Boi, Djamal Merad

► To cite this version:

Raphael Abele, Redouane El Moubtahij, Daniele Fronte, Pierre-Yvan Liardet, Jean-Luc Damoiseaux, et al.. FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy. IEEE Photonics Journal, 2019, 11 (5), pp.1-17. 10.1109/JPHOT.2019.2940689. hal-02315710

HAL Id: hal-02315710

https://amu.hal.science/hal-02315710v1

Submitted on 23 Mar 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Volume 11, Number 5, October 2019

Raphael Abele

Redouane El Moubtahij

Daniele Fronte

Pierre-Yvan Liardet

Jean-Luc Damoiseaux

Jean-Marc Boi

Djamal Merad

<figure><img src="image_3.png" /></figure>

DOI: 10.1109/JPHOT.2019.2940689
<figure><img src="image_2.png" /></figure>

# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Volume 11, Number 5, October 2019

Raphael Abele

Redouane El Moubtahij

Daniele Fronte

Pierre-Yvan Liardet

Jean-Luc Damoiseaux

Jean-Marc Boi

Djamal Merad

<figure><img src="image_3.png" /></figure>

DOI: 10.1109/JPHOT.2019.2940689
# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Raphael Abele<fnref n="1" />,<fnref n="2" />, Redouane El Moubtahij,<fnref n="1" /> Daniele Fronte,<fnref n="2" />
Pierre-Yvan Liardet,<fnref n="2" /> Jean-Luc Damoiseaux<fnref n="1" />,<fnref n="2" />, Jean-Marc Boi,<fnref n="1" />
and Djamal Merad<fnref n="1" />

DOI:10.1109/JPHOT.2019.2940689

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
https://creativecommons.org/licenses/by/4.0/

Manuscript received July 1, 2019; revised September 2, 2019; accepted September 5, 2019. Date of publication September 1, 2019; date of current version October 2, 2019. This work was supported in part by the Association Nationale de la Recherche et de la Technologie (ANRT). Corresponding author: Raphael Abele (e-mail: raphael.abele@st.com).

**Abstract:** This paper proposes an innovative autofocus method to ensure the image of an integrated circuit is correctly in focus under an infrared microscope. It discusses the difficulties inherent to the optical system and explores several inefficient methods used for natural scenes. It will also present a Focus Metric based on Polynomial Decomposition (FMPOD) adapted to our context. This approach relies on analyzing the projection of images on an orthonormal polynomial basis. Its robustness is tested using different magnifying lenses in addition to multiple distortions. In conclusion, we will demonstrate how this novel approach outperforms existing methods related to our work environment.

**Index Terms:** Infrared imaging, Microscopy, Focusing, Focus criterion, Image quality, Image decomposition, Polynomial decomposition.

## List of Acronyms

<table><tr><td>AF</td><td>Autofocus 2-7, 11, 12</td></tr><tr><td>COW</td><td>Custom Orthogonal Wavelet 4, 11, 12</td></tr><tr><td>DWT</td><td>Discrete Wavelet Transform 4, 7, 11, 12</td></tr><tr><td>FISH</td><td>Fast Index SHarpness 11, 12</td></tr><tr><td>FM</td><td>Focus Metric 3, 4, 10-12</td></tr><tr><td>FMPOD</td><td>Focus Metric based on POlynomial Decomposition 1, 7, 10-12</td></tr><tr><td>HVS</td><td>Human Visual System 3, 7</td></tr><tr><td>IC</td><td>Integrated Circuit 1, 2, 4, 6, 8, 11, 12</td></tr><tr><td>IQA</td><td>Image Quality Assessment 3, 7</td></tr><tr><td>IR</td><td>Infrared 1, 4, 5, 7, 8, 10, 11</td></tr><tr><td>PD</td><td>Polynomial Decomposition 7, 10</td></tr><tr><td>RR</td><td>Reduced-Reference 7</td></tr><tr><td>SD</td><td>Standard Deviation 4, 10</td></tr><tr><td>SOI</td><td>Surface Of Interest 1, 4-7, 11</td></tr><tr><td>SWIR</td><td>Short-Wave Infrared 4, 5</td></tr></table>

<footnote n="1">¹Laboratoire d'Informatique et Systemes, Aix-Marseille University, Marseille, 13288, France</footnote>

<footnote n="2">²STMicroelectronics, Rousset 13106, France</footnote>
# FMPOD: A Novel Focus Metric Based on Polynomial Decomposition for Infrared Microscopy

Raphael Abele<fnref n="1" />,<fnref n="2" />, Redouane El Moubtahij,<fnref n="1" /> Daniele Fronte,<fnref n="2" />
Pierre-Yvan Liardet,<fnref n="2" /> Jean-Luc Damoiseaux<fnref n="1" />,<fnref n="2" />, Jean-Marc Boi,<fnref n="1" />
and Djamal Merad<fnref n="1" />

DOI:10.1109/JPHOT.2019.2940689

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see
https://creativecommons.org/licenses/by/4.0/

Manuscript received July 1, 2019; revised September 2, 2019; accepted September 5, 2019. Date of publication September 1, 2019; date of current version October 2, 2019. This work was supported in part by the Association Nationale de la Recherche et de la Technologie (ANRT). Corresponding author: Raphael Abele (e-mail: raphael.abele@st.com).

**Abstract:** This paper proposes an innovative autofocus method to ensure the image of an integrated circuit is correctly in focus under an infrared microscope. It discusses the difficulties inherent to the optical system and explores several inefficient methods used for natural scenes. It will also present a Focus Metric based on Polynomial Decomposition (FMPOD) adapted to our context. This approach relies on analyzing the projection of images on an orthonormal polynomial basis. Its robustness is tested using different magnifying lenses in addition to multiple distortions. In conclusion, we will demonstrate how this novel approach outperforms existing methods related to our work environment.

**Index Terms:** Infrared imaging, Microscopy, Focusing, Focus criterion, Image quality, Image decomposition, Polynomial decomposition.

## List of Acronyms

<table><tr><td>AF</td><td>Autofocus 2-7, 11, 12</td></tr><tr><td>COW</td><td>Custom Orthogonal Wavelet 4, 11, 12</td></tr><tr><td>DWT</td><td>Discrete Wavelet Transform 4, 7, 11, 12</td></tr><tr><td>FISH</td><td>Fast Index SHarpness 11, 12</td></tr><tr><td>FM</td><td>Focus Metric 3, 4, 10-12</td></tr><tr><td>FMPOD</td><td>Focus Metric based on POlynomial Decomposition 1, 7, 10-12</td></tr><tr><td>HVS</td><td>Human Visual System 3, 7</td></tr><tr><td>IC</td><td>Integrated Circuit 1, 2, 4, 6, 8, 11, 12</td></tr><tr><td>Q&A</td><td>Image Quality Assessment 3, 7</td></tr><tr><td>IR</td><td>Infrared 1, 4, 5, 7, 8, 10, 11</td></tr><tr><td>PD</td><td>Polynomial Decomposition 7, 10</td></tr><tr><td>RR</td><td>Reduced-Reference 7</td></tr><tr><td>SD</td><td>Standard Deviation 4, 10</td></tr><tr><td>SOI</td><td>Surface Of Interest 1, 4-7, 11</td></tr><tr><td>SWIR</td><td>Short-Wave Infrared 4, 5</td></tr></table>

<footnote n="1">¹Laboratoire d'Informatique et Systemes, Aix-Marseille University, Marseille, 13288, France</footnote>

<footnote n="2">²STMicroelectronics, Rousset 13106, France</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 1. Concept mapping of three materials composing our IC.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Minimalist optical system; the image on plane 2 (green) is well focused on the optical sensor whereas the image on plane 1 is not.</figcaption></figure>

WDR Wide Dynamic Range 5, 6, 11

WPT Wavelet Packet Transform 4, 7, 11, 12

WT Wavelet Transform 4

# 1. Introduction

Integrated Circuits (ICs)<fnref n="1" /> are electronic components whose applications seem unlimited: they are used in many fields such as wearable technologies and IoT (Internet of Objects). In order to protect the privacy of users, some ICs need to be secured, and their security needs to be validated. Among the tasks to characterize a secure IC is the study of its behavior following a physical disruption. Such a disruption may be obtained by using a laser beam to target specific areas of the IC's internal structures. The accuracy of the shot relies on the precision of the laser power calibration and 3D positioning inside the IC. In our study, an IC is considered as made of three layers of different materials: (1) the silicon, (2) a conductor material forming the conductive tracks, and (3) an electrical insulator, as illustrated in Fig. 1.

The internal structures of the IC are visible on the surface of the conductive tracks, which is the Surface Of Interest (SOI). The laser beam is positioned three-dimensionally on the SOI using an Infrared (IR) camera that can see through the silicon layer. In our work, the optical system and the laser source are interdependent, so the optical adjustment implies accurately adjusting the laser beam's position (i.e. their focal point). Usually, these adjustments are made manually by skilled personnel, which raises two problems: time loss and non-reproducibility of the characterization (because of potential imprecision). An Autofocus (AF) system will solve the problems by automating one dimension of the 3D positioning process, that is, the automation of the focus research related to the Z-axis. This paper purpose is to provide a robust AF solution for low-contrast infrared images in the context of integrated circuits vision. The following section makes a review of the related works (Section 2). Section 3 describes the context of our study together with the related constraints that highlight the need of a unique method adapted to our use.

# 2. Around the Autofocus

The Autofocus mechanism is a deterministic algorithm used to compute the lens position such that the system is “well focused” on the scene/object of interest (see Fig. 2). As a consequence, the image of the scene or object is the sharpest. Two types of AF approaches exist: active and passive [1].

<footnote n="1">¹A glossary is present at the end of this paper.</footnote>
<figure><img src="image_4.png" /><figcaption>Fig. 1. Concept mapping of three materials composing our IC.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 2. Minimalist optical system; the image on plane 2 (green) is well focused on the optical sensor whereas the image on plane 1 is not.</figcaption></figure>

WDR Wide Dynamic Range 5, 6, 11

WPT Wavelet Packet Transform 4, 7, 11, 12

WT Wavelet Transform 4

# 1. Introduction

Integrated Circuits (ICs)<fnref n="1" /> are electronic components whose applications seem unlimited: they are used in many fields such as wearable technologies and IoT (Internet of Objects). In order to protect the privacy of users, some ICs need to be secured, and their security needs to be validated. Among the tasks to characterize a secure IC is the study of its behavior following a physical disruption. Such a disruption may be obtained by using a laser beam to target specific areas of the IC's internal structures. The accuracy of the shot relies on the precision of the laser power calibration and 3D positioning inside the IC. In our study, an IC is considered as made of three layers of different materials: (1) the silicon, (2) a conductor material forming the conductive tracks, and (3) an electrical insulator, as illustrated in Fig. 1.

The internal structures of the IC are visible on the surface of the conductive tracks, which is the Surface Of Interest (SOI). The laser beam is positioned three-dimensionally on the SOI using an Infrared (IR) camera that can see through the silicon layer. In our work, the optical system and the laser source are interdependent, so the optical adjustment implies accurately adjusting the laser beam's position (i.e. their focal point). Usually, these adjustments are made manually by skilled personnel, which raises two problems: time loss and non-reproducibility of the characterization (because of potential imprecision). An Autofocus (AF) system will solve the problems by automating one dimension of the 3D positioning process, that is, the automation of the focus research related to the Z-axis. This paper purpose is to provide a robust AF solution for low-contrast infrared images in the context of integrated circuits vision. The following section makes a review of the related works (Section 2). Section 3 describes the context of our study together with the related constraints that highlight the need of a unique method adapted to our use.

# 2. Around the Autofocus

The Autofocus mechanism is a deterministic algorithm used to compute the lens position such that the system is “well focused” on the scene/object of interest (see Fig. 2). As a consequence, the image of the scene or object is the sharpest. Two types of AF approaches exist: active and passive [1].

<footnote n="1">¹A glossary is present at the end of this paper.</footnote>
## 2.1 Active AF Approaches

An active approach depends on an additional system measuring the distance from the lens to the scene. Knowing the optics-to-object distance and the system focal length, the correct lens position is estimated to obtain a focused image.

Early methods use ultrasonic sound waves or infrared light to measure distance by auxiliary signal measurement [2]. A recent technology based on time-of-flight camera creates a complete depths map of a scene, or more precisely a map of each pixel of the camera sensor [3], [4]; innovations in this technology are still being studied [5]. A different system uses a multi-lens array positioned in front of the sensor; the resulting data is then post-processed to post-focus in the scene in any image location. This concept called light-field was conceptually introduced by Gershun [6] and used for an autofocus system by Stauffer [7]. It is still being studied today [8]–[10] and was extended to the microscopy vision as described by Levoy et al. [11]. It also may have a great impact on 3D image acquisitions [12]. An innovative system by Bathe-Peters et al. [13] for microscopy proposes a fully automated active AF system, regardless the objective in use. It uses an electrically tunable lens to modulate the focal distance of the optical system. This lens is stimulated by a photo detector measuring the stability of a pointing laser beam reflected by the scene. All above technologies allow to overcome the need of autofocusing the scene; however, in our study we dispose of the optical system without such technologies or any external measurement tool. Then we only consider passive approaches.

## 2.2 Passive AF Approaches

A passive approach relies on image analysis to determine the most relevant image of the scene. It means that images should be evaluated as the human visual system does. If correctly identified using Image Quality Assessment (IQA), it can determine the optimal lens position. We call this a Focus Metric (FM). Since the Human Visual System (HVS) is the most reliable tool for IQA, the main difficulty is the interpretation of human subjectivity through algorithms following objective rules and criteria [14]. Some subjective methods attempt to emulate the human visual system, but they are time-consuming and usually non-deterministic. Objective IQA methods attempt to be correlated with the HVS. Depending on the amount of available information from the image source, the metrics are categorized as:

* Full-Reference based if an original, non-distorted image is available [15],

* No-Reference based otherwise [16],

* Reduced-Reference based if just some of the original image features are known.

[17] includes examples. Reduced-Reference-IQA methods are usually guided/optimized No-Reference-IQA methods. In the AF context, both the No-Reference- and the Reduced-Reference-IQA methods are used, since the quality criterion is hypothetical. Images are ranked according to this criterion, and the best-ranked image is associated with the correct lens position. A possibility is to pool together different objective metrics to predict the quality score, namely the Mean Opinion Score (MOS) [18], but it may require heavy computational work to process several criteria.

In our study, we consider passive AF approaches: despite the special illuminating system necessary for images acquisition, no extra device is available for any useful measurement for an AF process. Since AF algorithms rely on specific image analysis, it is hard to establish a generalized approach. Several methods exist in the literature, each one adapted to its specific context such as images of natural scenes, low-contrast images, microscopy, digital holography, Synthetic Aperture Radar (SAR) and so on. The first few AF algorithms are based on the sharpness concept of an image, which may be solved as a gradient maximization, edge transition-width minimization or histogram entropy minimization [19]–[21]. Some applications have compared the gradient problem to a sparsity problem around edges; this is the case in most AF solutions deployed recently in holography imaging [22]–[24].

In fact all the existing AF algorithms use the most appropriate approach compatible with their context. The focus criteria from the previously cited papers depends on the spatial repartition of the
## 2.1 Active AF Approaches

An active approach depends on an additional system measuring the distance from the lens to the scene. Knowing the optics-to-object distance and the system focal length, the correct lens position is estimated to obtain a focused image.

Early methods use ultrasonic sound waves or infrared light to measure distance by auxiliary signal measurement [2]. A recent technology based on *time-of-flight* camera creates a complete depths map of a scene, or more precisely a map of each pixel of the camera sensor [3], [4]; innovations in this technology are still being studied [5]. A different system uses a multi-lens array positioned in front of the sensor; the resulting data is then post-processed to post-focus in the scene in any image location. This concept called *light-field* was conceptually introduced by Gershun [6] and used for an autofocus system by Stauffer [7]. It is still being studied today [8]–[10] and was extended to the microscopy vision as described by Levoy *et al.* [11]. It also may have a great impact on 3D image acquisitions [12]. An innovative system by Bathe-Peters *et al.* [13] for microscopy proposes a fully automated active AF system, regardless the objective in use. It uses an electrically tunable lens to modulate the focal distance of the optical system. This lens is stimulated by a photo detector measuring the stability of a pointing laser beam reflected by the scene. All above technologies allow to overcome the need of autofocusing the scene; however, in our study we dispose of the optical system without such technologies or any external measurement tool. Then we only consider passive approaches.

## 2.2 Passive AF Approaches

A passive approach relies on image analysis to determine the most relevant image of the scene. It means that images should be evaluated as the human visual system does. If correctly identified using Image Quality Assessment (IQA), it can determine the optimal lens position. We call this a Focus Metric (FM). Since the Human Visual System (HVS) is the most reliable tool for IQA, the main difficulty is the interpretation of human subjectivity through algorithms following objective rules and criteria [14]. Some subjective methods attempt to emulate the human visual system, but they are time-consuming and usually non-deterministic. Objective IQA methods attempt to be correlated with the HVS. Depending on the amount of available information from the image source, the metrics are categorized as:

* Full-Reference based if an original, non-distorted image is available [15],

* No-Reference based otherwise [16],

* Reduced-Reference based if just some of the original image features are known.

[17] includes examples. Reduced-Reference-IQA methods are usually guided/optimized No-Reference-IQA methods. In the AF context, both the No-Reference- and the Reduced-Reference-IQA methods are used, since the quality criterion is hypothetical. Images are ranked according to this criterion, and the best-ranked image is associated with the correct lens position. A possibility is to pool together different objective metrics to predict the quality score, namely the Mean Opinion Score (MOS) [18], but it may require heavy computational work to process several criteria.

In *our study*, we consider passive AF approaches: despite the special illuminating system necessary for images acquisition, no extra device is available for any useful measurement for an AF process. Since AF algorithms rely on specific image analysis, it is hard to establish a generalized approach. Several methods exist in the literature, each one adapted to its specific context such as images of natural scenes, low-contrast images, microscopy, digital holography, Synthetic Aperture Radar (SAR) and so on. The first few AF algorithms are based on the sharpness concept of an image, which may be solved as a gradient maximization, edge transition-width minimization or histogram entropy minimization [19]–[21]. Some applications have compared the gradient problem to a sparsity problem around edges; this is the case in most AF solutions deployed recently in holography imaging [22]–[24].

In fact all the existing AF algorithms use the most appropriate approach compatible with their context. The focus criteria from the previously cited papers depends on the spatial repartition of the
pixels in the images and their relatively near neighborhood. The focus metrics can rely on various statistics of order 1, 2 or higher, since each one may point out an image feature [25]. Moreover, image statistics can be declined into spatial, frequency and time-frequency domains. In the context of biological microscopy, Zhang *et al.* [26] compared the performance and accuracy of sixteen AF methods: histogram, intensity, statistic, derivative and transform-based. They found that the Absolute Tenengrad algorithm has the best performance in accuracy for its specifics. To evaluate the Absolute Tenengrad, the image gradient is evaluated and the gradients of highest amplitude are considered to compute a sharpness metric. The images studied are contrasted enough to make the metric relatively robust for their application, which is not our case (see Section 3).

Fonseca *et al.* [27] have made an overall comparison among several focus measures, usually used for natural scenes, adapted to a holography context. For the focus measures, the view is reduced to a standard 2D gray-level image. The measures are grouped into spatial-based category on one side and a spectral-based one on the other side. This comparison in terms of unimodality, accuracy, resolution, and computational cost highlighted that each approach has its pros and cons. In any case, they are all adapted to natural scene images, thus they are unadapted to our study.

In [28], an AF algorithm is proposed for microscopy. The focus criterion is based on a Bayes-Spectral-Entropy measure using the Discrete Cosinus Transform. This measure on the image spectrum is computed to determine whether an image is out of focus (blurred). In their study, the object of interest is a calibration chart illuminated by natural light and contrasted enough to facilitate microscope calibration.

In [29], the AF measure is based on both the Medium Frequency Discrete Cosine Transform and the Discrete Cosinus Transform. Their motivation: medium-frequency components of an image are more stable and less susceptible to noise than the high-and low-frequency portions. This approach makes sense in our study since we face a large amount of noise in our images, even more than in their study. Similarly in [30], the authors distinguish high from medium frequencies extracted using the cosine transform. Our approach, proposed in Section 4, also makes a distinction between high, mid and low frequencies.

The Wavelet Packet Transform (WPT) is used as FM in [31]. As previously stated, the WPT strength against the Wavelet Transform (WT) is its ability to extract high frequencies from a signal. The primary assumption of the author is: one particularity of the focused image is its high frequencies. The WPT coefficients representing horizontal, vertical and diagonal data are weighted respectively by horizontal, vertical and diagonal image gradients, and summed together to constitute the focus metric. In their experiment [31], the object of interest is a high-temperature blackbody. The optical system is composed of an infrared microscope which magnifies the view, which is then captured by an IR array. This mechanism gives very sharp, almost binary, images when the optical system is focused on the target. In this case, the use of the WPT is well adapted and gives the expected results: the more focused is the system, the higher the local gradients and the more high frequencies.

In a similar way, but based on the Discrete Wavelet Transform (DWT), *Vu and Chandler* developed an image quality assessment [32] to quantify the sharpness of an image by analyzing the coefficients of the DWT using the Cohen-Daubechies-Fauveau 9/7 wavelet. More weight is given to the higher frequencies to compute the quality metric. Recently, we also used the DWT coefficients for the autofocusing purpose, in infrared microscopy [33]. The Standard Deviation (SD) of specific DWT coefficients is considered to compute the focus metric. The context was the same as in the present study. As highlighted in Section 3.2, the highest frequencies in our images may be impacted by the thermal noise and the textural grain, especially when using higher magnifying lenses. Then the limitation of wavelets in term of lack of high frequencies is not considered as a binding problem, since it allows skipping irrelevant information related to material texture and thermal noise. Moreover, since the wavelet choice is of main priority [34], the Custom Orthogonal Wavelet (COW) is designed to fit the relevant information of their IR images, and the FM correspond to the SD of the vertical and horizontal wavelet coefficients.

Finally, each AF algorithm uses a well justified metric, related to a particular context. In the present study, since we face the same context as in [33] and its resultant constraints, we will
pixels in the images and their relatively near neighborhood. The focus metrics can rely on various statistics of order 1, 2 or higher, since each one may point out an image feature [25]. Moreover, image statistics can be declined into spatial, frequency and time-frequency domains. In the context of biological microscopy, Zhang *et al.* [26] compared the performance and accuracy of sixteen AF methods: histogram, intensity, statistic, derivative and transform-based. They found that the Absolute Tenengrad algorithm has the best performance in accuracy for its specifics. To evaluate the Absolute Tenengrad, the image gradient is evaluated and the gradients of highest amplitude are considered to compute a sharpness metric. The images studied are contrasted enough to make the metric relatively robust for their application, which is not our case (see Section 3).

Fonseca *et al.* [27] have made an overall comparison among several focus measures, usually used for natural scenes, adapted to a holography context. For the focus measures, the view is reduced to a standard 2D gray-level image. The measures are grouped into spatial-based category on one side and a spectral-based one on the other side. This comparison in terms of unimodality, accuracy, resolution, and computational cost highlighted that each approach has its pros and cons. In any case, they are all adapted to natural scene images, thus they are unadapted to our study.

In [28], an AF algorithm is proposed for microscopy. The focus criterion is based on a Bayes-Spectral-Entropy measure using the Discrete Cosinus Transform. This measure on the image spectrum is computed to determine whether an image is out of focus (blurred). In their study, the object of interest is a calibration chart illuminated by natural light and contrasted enough to facilitate microscope calibration.

In [29], the AF measure is based on both the Medium Frequency Discrete Cosine Transform and the Discrete Cosinus Transform. Their motivation: medium-frequency components of an image are more stable and less susceptible to noise than the high-and low-frequency portions. This approach makes sense in our study since we face a large amount of noise in our images, even more than in their study. Similarly in [30], the authors distinguish high from medium frequencies extracted using the cosine transform. Our approach, proposed in Section 4, also makes a distinction between high, mid and low frequencies.

The Wavelet Packet Transform (WPT) is used as FM in [31]. As previously stated, the WPT strength against the Wavelet Transform (WT) is its ability to extract high frequencies from a signal. The primary assumption of the author is: one particularity of the focused image is its high frequencies. The WPT coefficients representing horizontal, vertical and diagonal data are weighted respectively by horizontal, vertical and diagonal image gradients, and summed together to constitute the focus metric. In their experiment [31], the object of interest is a high-temperature blackbody. The optical system is composed of an infrared microscope which magnifies the view, which is then captured by an IR array. This mechanism gives very sharp, almost binary, images when the optical system is focused on the target. In this case, the use of the WPT is well adapted and gives the expected results: the more focused is the system, the higher the local gradients and the more high frequencies.

In a similar way, but based on the Discrete Wavelet Transform (DWT), *Vu and Chandler* developed an image quality assessment [32] to quantify the sharpness of an image by analyzing the coefficients of the DWT using the Cohen-Daubechies-Fauveau 9/7 wavelet. More weight is given to the higher frequencies to compute the quality metric. Recently, we also used the DWT coefficients for the autofocusing purpose, in infrared microscopy [33]. The Standard Deviation (SD) of specific DWT coefficients is considered to compute the focus metric. The context was the same as in the present study. As highlighted in Section 3.2, the highest frequencies in our images may be impacted by the thermal noise and the textural grain, especially when using higher magnifying lenses. Then the limitation of wavelets in term of lack of high frequencies is not considered as a binding problem, since it allows skipping irrelevant information related to material texture and thermal noise. Moreover, since the wavelet choice is of main priority [34], the Custom Orthogonal Wavelet (COW) is designed to fit the relevant information of their IR images, and the FM correspond to the SD of the vertical and horizontal wavelet coefficients.

Finally, each AF algorithm uses a well justified metric, related to a particular context. In the present study, since we face the same context as in [33] and its resultant constraints, we will
<figure><img src="image_6.png" /><figcaption>Fig. 3. Infrared wavelength in the electromagnetic spectrum.</figcaption></figure>

compare our new approach against the earlier wavelet-based one. Firstly, the context is presented in the Section 3, which highlights the inefficiency of the more general-purpose focus criteria in regards to natural scenes. Then the new FM is proposed in Section 4. Several AF results are reported in Section 5 and compared with three other methods including the method proposed in [33].

# 3. Work Environment

## 3.1 Available Tools

Observation of the SOI under the silicon layer is possible thanks to its optical properties. Indeed, around the wavelength 1 µm and higher, the silicon absorption coefficient is not significant [35], [36]. In other words, silicon is “transparent” to light with wavelengths approximately around 1 µm and above (such as IR light).

Our motorized optical system provided by AlphaNov is composed of:

* an uncooled IR camera

* an optical microscope with 4 magnifying lenses (2.5x, 5x, 20x and 50x) allowing us to observe the IC (approximately 5 mm²) and its internal structures (micro-metric scale)

Three types of IR cameras are available on the market: Long-Wave (LWIR), Mid-Wave (MWIR) and Short-Wave Infrared (SWIR). On one hand, MWIR and LWIR sensors detect thermal emissions from objects, and are efficient when these are warmer than their surroundings. On the other hand, SWIR cameras use the reflected light, much like the slightly shorter wavelengths of the visible spectrum (see Fig. 3). In our experiment we worked with the latter type of camera.

With the capabilities of our SWIR camera, we are able to see the SOI through the silicon layer. To propose an automated method for focusing on the SOI, the algorithm must take in account some specifics arising from the camera and from the SOI (see Section 3.2).

## 3.2 Constraints and Specific Features

### 3.2.1 About the Camera
The IR camera sensor is an InGaAs, sensitive to a large IR wavelength range (900 nm to 1700 nm); a Wide Dynamic Range (WDR) is natively embedded in the camera and automatically adapts the contrast level to have the best image as possible on the fly. The infrared sensor is uncooled, that is, thermally not insulated; and as such it is sensitive to external thermal interference. Then the thermal instability of the environment produces noise, which affects the image acquisition; see Fig. 4 as an example.

### 3.2.2 About the Object of Interest
Two distinct surfaces are considered for the AF algorithm: the silicon and the SOI. As said in Section 3.1, silicon is transparent to our IR light and should not be visible. However, in many cases, contaminants (e.g. dusts) are ingrained on the silicon, which reveals the surface to any light. This implies a possible conflict of interest, considering the sharpness criterion: the silicon surface with its contaminants may be sharper than the SOI. This point
<figure><img src="image_6.png" /><figcaption>Fig. 3. Infrared wavelength in the electromagnetic spectrum.</figcaption></figure>

compare our new approach against the earlier wavelet-based one. Firstly, the context is presented in the Section 3, which highlights the inefficiency of the more general-purpose focus criteria in regards to natural scenes. Then the new FM is proposed in Section 4. Several AF results are reported in Section 5 and compared with three other methods including the method proposed in [33].

# 3. Work Environment

## 3.1 Available Tools

Observation of the SOI under the silicon layer is possible thanks to its optical properties. Indeed, around the wavelength 1 µm and higher, the silicon absorption coefficient is not significant [35], [36]. In other words, silicon is “transparent” to light with wavelengths approximately around 1 µm and above (such as IR light).

Our motorized optical system provided by AlphaNov is composed of:

* an uncooled IR camera

* an optical microscope with 4 magnifying lenses (2.5x, 5x, 20x and 50x) allowing us to observe the IC (approximately 5 mm²) and its internal structures (micro-metric scale)

Three types of IR cameras are available on the market: Long-Wave (LWIR), Mid-Wave (MWIR) and Short-Wave Infrared (SWIR). On one hand, MWIR and LWIR sensors detect thermal emissions from objects, and are efficient when these are warmer than their surroundings. On the other hand, SWIR cameras use the reflected light, much like the slightly shorter wavelengths of the visible spectrum (see Fig. 3). In our experiment we worked with the latter type of camera.

With the capabilities of our SWIR camera, we are able to see the SOI through the silicon layer. To propose an automated method for focusing on the SOI, the algorithm must take in account some specifics arising from the camera and from the SOI (see Section 3.2).

## 3.2 Constraints and Specific Features

3.2.1 About the Camera: The IR camera sensor is an InGaAs, sensitive to a large IR wavelength range (900 nm to 1700 nm); a Wide Dynamic Range (WDR) is natively embedded in the camera and automatically adapts the contrast level to have the best image as possible on the fly. The infrared sensor is uncooled, that is, thermally not insulated; and as such it is sensitive to external thermal interference. Then the thermal instability of the environment produces noise, which affects the image acquisition; see Fig. 4 as an example.

3.2.2 About the Object of Interest: Two distinct surfaces are considered for the AF algorithm: the silicon and the SOI. As said in Section 3.1, silicon is transparent to our IR light and should not be visible. However, in many cases, contaminants (e.g. dusts) are ingrained on the silicon, which reveals the surface to any light. This implies a possible conflict of interest, considering the sharpness criterion: the silicon surface with its contaminants may be sharper than the SOI. This point
<figure><img src="image_7.png" /><figcaption>Fig. 4. An example of colored images acquired with cooled and uncooled infrared cameras (www.flir.com).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 5. Images acquired at several focal point positions (green points) according to the Z-axis. Component out of focus (left), focused on the silicon surface (middle) and on the conductive track surface (right).</figcaption></figure>

underlines that sharpness is not, in practice, the appropriate criterion to exclusively characterize an SOI. Fig. 5 shows to the notable states during the AF process.

We may compare the conductive layer of an IC to a city, and its internal structures to buildings. As shown in Fig. 6, everything is orthogonally positioned, in regard to the rectangular shape of the component. This criterion is a good candidate for our AF algorithm.

However, a difficulty arises from the SOI image texture. An IC is built by microscopic material deposition, which implies a textural grain visible with a high magnifying lens (20x and 50x). Considering the SOI images, this grain can be compared to noise in the images: structures are clusters of points, more or less dense (illustrated in Fig. 7). In the worst case, it becomes difficult to detect the structure's boundaries and the images may require preprocessing to be correctly developed. Moreover, depending on the quality of the deposit material used for the IC's design, its quality and/or its amount, the conductive layer of the IC may reflect the IR light in a small proportion. It could leads to a low contrasted view of the SOI, in some case lower than an out-of-focus view because of the WDR embedded system.

To sum up, we may state that:

1) Sharpness is not a good criterion to make the difference between the silicon surface and the SOI.
<figure><img src="image_7.png" /><figcaption>Fig. 4. An example of colored images acquired with cooled and uncooled infrared cameras (www.flir.com).</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 5. Images acquired at several focal point positions (green points) according to the Z-axis. Component out of focus (left), focused on the silicon surface (middle) and on the conductive track surface (right).</figcaption></figure>

underlines that sharpness is not, in practice, the appropriate criterion to exclusively characterize an SOI. Fig. 5 shows to the notable states during the AF process.

We may compare the conductive layer of an IC to a city, and its internal structures to buildings. As shown in Fig. 6, everything is orthogonally positioned, in regard to the rectangular shape of the component. This criterion is a good candidate for our AF algorithm.

However, a difficulty arises from the SOI image texture. An IC is built by microscopic material deposition, which implies a textural grain visible with a high magnifying lens (20x and 50x). Considering the SOI images, this grain can be compared to noise in the images: structures are clusters of points, more or less dense (illustrated in Fig. 7). In the worst case, it becomes difficult to detect the structure's boundaries and the images may require preprocessing to be correctly developed. Moreover, depending on the quality of the deposit material used for the IC's design, its quality and/or its amount, the conductive layer of the IC may reflect the IR light in a small proportion. It could leads to a low contrasted view of the SOI, in some case lower than an out-of-focus view because of the WDR embedded system.

To sum up, we may state that:

1) Sharpness is not a good criterion to make the difference between the silicon surface and the SOI.
<figure><img src="image_9.png" /><figcaption>Fig. 6. A simple representation of an IC and its internal structures.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 7. Two lines delimiting the structure's boundary (50x).</figcaption></figure>

2) On the SOI, the structure's horizontal and vertical boundaries are visible.

3) Because of textural grain and noise, topological information is not relevant (e.g. line detection). Moreover, this may impact the image's highest frequencies.

4) The AF method must be effective for each magnifying lens (2.5x, 5x, 20x and 50x).

5) The method must be effective regardless of the contrast in the SOI image.

Since we know which features are expected in the targeted image (2), the expected IQA is qualified as Reduced-Reference (RR)-based. Moreover, for the sake of a reduced runtime, preprocessing such as image denoising has to be avoided (3); then transform-based approaches are considered and two focus criteria are presented in the following section.

In the following, a new focus criterion is presented in Section 4, considering the polynomial decomposition whose choice is also discussed. The experimental results of this criterion applied to our AF algorithm is discussed in Section 5.

## 4. Novel Focus Metric Based on Polynomial Decomposition (FMPOD)

As previously pointed out in Section 2.2, many different statistics can reflect an image feature. These statistics can be computed in spatial, frequency and time-frequency domains. To go from the image dimension to the frequency or time-frequency domains, many transforms exist starting with the Discrete Fourier Transform to the Stockwell Transform [37], [38] all the way through to the Polynomial Decomposition (PD) [39] and the DWT [40]. Each one has its strengths and weaknesses in signal analysis, and over the years new transforms were adopted, mainly derived from the
<figure><img src="image_9.png" /><figcaption>Fig. 6. A simple representation of an IC and its internal structures.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Fig. 7. Two lines delimiting the structure's boundary (50x).</figcaption></figure>

2) On the SOI, the structure's horizontal and vertical boundaries are visible.

3) Because of textural grain and noise, topological information is not relevant (e.g. line detection). Moreover, this may impact the image's highest frequencies.

4) The AF method must be effective for each magnifying lens (2.5x, 5x, 20x and 50x).

5) The method must be effective regardless of the contrast in the SOI image.

Since we know which features are expected in the targeted image (2), the expected IQA is qualified as Reduced-Reference (RR)-based. Moreover, for the sake of a reduced runtime, preprocessing such as image denoising has to be avoided (3); then transform-based approaches are considered and two focus criteria are presented in the following section.

In the following, a new focus criterion is presented in Section 4, considering the polynomial decomposition whose choice is also discussed. The experimental results of this criterion applied to our AF algorithm is discussed in Section 5.

## 4. Novel Focus Metric Based on Polynomial Decomposition (FMPOD)

As previously pointed out in Section 2.2, many different statistics can reflect an image feature. These statistics can be computed in spatial, frequency and time-frequency domains. To go from the image dimension to the frequency or time-frequency domains, many transforms exist starting with the Discrete Fourier Transform to the Stockwell Transform [37], [38] all the way through to the Polynomial Decomposition (PD) [39] and the DWT [40]. Each one has its strengths and weaknesses in signal analysis, and over the years new transforms were adopted, mainly derived from the
wavelet transform. The undecimated wavelet transform [34], the curvelet tranform [41] and the linelet transform [42] are recent methods inspired from the wavelet transform.

The PD of an image was presented by Eden et al. [39] and has similarities with WPT for defined scale. They also pointed out that local polynomial representation appears to be particularly suited for feature extraction and may be adaptable to such operations as edge detection or the measurement and classification of texture properties. In [43] the PD was successfully used to detect singularity points and allowed an accurate texture description of images in [44] and [45].

In [46] the possibility that the HVS distinguishing the visual information as elementary orthogonal harmonic from the signal is emitted. Indeed, the discrete values of a numerical image may be considered as the part of a continuous complex function which can be approximated with functions. For instance:

* Legendre and Gauss have used the polynomial functions to approximate a complex one,

* Chebychev has developed the concept of uniform approximation,

* Weierstrass has proven that polynomials may approximate in a interval any real continuous function,

* Taylor series has been largely used to approximate the value in the neighborhood of a given point.

It makes sense that the PD of an image allows a mathematically realistic approximation of an image, and we decided to use the polynomial image decomposition to analyze our images and address our problem of IR microscopy AF. In the next sections we introduce the polynomial decomposition and its use in the FMPOD.

## 4.1 About the Polynomial Decomposition

In their study, R. Moubtahij et al. [47] use an orthonormal polynomial basis introduced by O. Kihl et al. [48] to make a multi-scale decomposition of images. An orthonormal polynomial basis is composed of polynomials P whose particularity comes from their scalar product. The scalar product of two functions $f_1$ and $f_2$ can be computed as:

$$ \langle f_1 | f_2 \rangle = \iint_{\Omega} f_1(x) f_2(x) \omega(x) dx \quad (1) $$

with $\forall x \in \Omega, \omega(x) \ge 0$, where $\langle \cdot | \cdot \rangle$ represent the scalar product, and $\omega$ is a weighting function. Let's consider the polynomials as formulated below:

$$ P_{d_1, d_2}(x) = \sum_{\substack{(d_1, d_2) \in [0; d]^2 \\ d_1 + d_2 \le d}} a_{d_1, d_2} x_1^{d_1} x_2^{d_2} \quad (2) $$

where $d_1$ and $d_2$ are respectively the degrees of variables $x_1$ and $x_2$, and $a_{d_1, d_2}$ is the group of real coefficients of the polynomial. The polynomial degree is given by $D = \sup(d_1, d_2)$. In the case of a polynomial basis, the scalar product of two polynomials from basis $B_D$ is defined as follow:

$$ \langle P_{d_1, d_2} | P_{l_1, l_2} \rangle_{\omega} = \begin{cases} 0 & \text{if } (d_1, d_2) \neq (l_1, l_2) \\ 1 & \text{if } (d_1, d_2) = (l_1, l_2) \end{cases} \quad (3) $$

The type of weighting function of the scalar product characterizes the type of polynomial basis. Some examples of such functions are reported in Table 1.

The approximation $\tilde{I}$ of an image $I$ through an orthonormal basis $B_D$ and relatively to the base type is:

$$ \tilde{I}(x) = \sum_{\substack{(d_1, d_2) \in [0; d]^2 \\ d_1 + d_2 \le d}} a_{d_1, d_2}(l) B_{d_1, d_2}(x) \quad (4) $$
wavelet transform. The undecimated wavelet transform [34], the curvelet tranform [41] and the linelet transform [42] are recent methods inspired from the wavelet transform.

The PD of an image was presented by Eden et al. [39] and has similarities with WPT for defined scale. They also pointed out that local polynomial representation appears to be particularly suited for feature extraction and may be adaptable to such operations as edge detection or the measurement and classification of texture properties. In [43] the PD was successfully used to detect singularity points and allowed an accurate texture description of images in [44] and [45].

In [46] the possibility that the HVS distinguishing the visual information as elementary orthogonal harmonic from the signal is emitted. Indeed, the discrete values of a numerical image may be considered as the part of a continuous complex function which can be approximated with functions. For instance:

* Legendre and Gauss have used the polynomial functions to approximate a complex one,

* Chebychev has developed the concept of uniform approximation,

* Weierstrass has proven that polynomials may approximate in a interval any real continuous function,

* Taylor series has been largely used to approximate the value in the neighborhood of a given point.

It makes sense that the PD of an image allows a mathematically realistic approximation of an image, and we decided to use the polynomial image decomposition to analyze our images and address our problem of IR microscopy AF. In the next sections we introduce the polynomial decomposition and its use in the FMPOD.

## 4.1 About the Polynomial Decomposition

In their study, R. Moubtahij et al. [47] use an orthonormal polynomial basis introduced by O. Kihl et al. [48] to make a multi-scale decomposition of images. An orthonormal polynomial basis is composed of polynomials P whose particularity comes from their scalar product. The scalar product of two functions $f_1$ and $f_2$ can be computed as:

$$ \langle f_1 | f_2 \rangle = \iint_{\Omega} f_1(x) f_2(x) \omega(x) dx \quad (1) $$

with $\forall x \in \Omega$, $\omega(x) \ge 0$, where $\langle \cdot | \cdot \rangle$ represent the scalar product, and $\omega$ is a weighting function. Let's consider the polynomials as formulated below:

$$ P_{d_1, d_2}(x) = \sum_{\substack{(d_1, d_2) \in [0; d]^2 \\ d_1 + d_2 \le d}} a_{d_1, d_2} x_1^{d_1} x_2^{d_2} \quad (2) $$

where $d_1$ and $d_2$ are respectively the degrees of variables $x_1$ and $x_2$, and $a_{d_1, d_2}$ is the group of real coefficients of the polynomial. The polynomial degree is given by $D = \sup(d_1, d_2)$. In the case of a polynomial basis, the scalar product of two polynomials from basis $B_D$ is defined as follow:

$$ \langle P_{d_1, d_2} | P_{l_1, l_2} \rangle_{\omega} = \begin{cases} 0 & \text{if } (d_1, d_2) \neq (l_1, l_2) \\ 1 & \text{if } (d_1, d_2) = (l_1, l_2) \end{cases} \quad (3) $$

The type of weighting function of the scalar product characterizes the type of polynomial basis. Some examples of such functions are reported in Table 1.

The approximation $\tilde{I}$ of an image $I$ through an orthonormal basis $B_D$ and relatively to the base type is:

$$ \tilde{I}(x) = \sum_{\substack{(d_1, d_2) \in [0; d]^2 \\ d_1 + d_2 \le d}} a_{d_1, d_2}(l) B_{d_1, d_2}(x) \quad (4) $$
<table><caption>TABLE 1
Few Weighting Functions ω Used for Different Families of Polynomials</caption><thead><tr><th>Polynomial</th><th>Interval</th><th>ω</th></tr></thead><tbody><tr><td>Legendre</td><td>[-1; 1]<sup>2</sup></td><td>1</td></tr><tr><td>Hermite</td><td>[-∞; +∞]<sup>2</sup></td><td>e<sup>-<sup>x<sub>1</sub><sup>2</sup> + x<sub>2</sub><sup>2</sup></sup> / 2</sup></td></tr><tr><td>Laguerre</td><td>[0; +∞]<sup>2</sup></td><td>e<sup>-<sup>(x<sub>1</sub> + x<sub>2</sub>)</sup> / 2</sup></td></tr><tr><td>Tchebychev1</td><td>[-1; 1]<sup>2</sup></td><td>1 / √<span style="text-decoration:overline;">(1 - x<sub>1</sub>)<sup>2</sup>(1 - x<sub>2</sub>)<sup>2</sup></span></td></tr><tr><td>Tchebychev2</td><td>[-1; 1]<sup>2</sup></td><td>√<span style="text-decoration:overline;">(1 - x<sub>1</sub>)<sup>2</sup>(1 - x<sub>2</sub>)<sup>2</sup></span></td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 8. Two decompositions of an IR image (50x): 2D DWT using the Haar wavelet (left) and polynomial decomposition using the Legendrian basis with a 2 × 2 support.</figcaption></figure>

where $a_{d_1, d_2}$ is the polynomial coefficients

$$a_{d_1, d_2}(I) = \frac{\langle I | P_{d_1, d_2} \rangle_{\omega}}{\langle P_{d_1, d_2} | P_{d_1, d_2} \rangle_{\omega}} \quad (5)$$

This scalar product can be calculated as an integral, sampled on a 2D grid whose points (or collocation points) may be equally spaced (discrete uniform distribution). The size of this grid will determine the approximation's fineness at each location in the image. An algorithm is proposed in [44] to approximate this projection.

Similarly to the wavelet decomposition approach, the choice of the approximation function is of utmost importance. As discussed in Section 3.2, our images have some specifics (noise, texture, horizontal/vertical components, etc.) and we take care to extract specific data from these images. Thus, the purpose is to choose the best function to decompose the complex signal representing the data of our images. In [47], the similarity is put forward between the Legendre polynomial discretized on a reduced range (2x2 approximation grid) and the Haar wavelet. This similitude is illustrated in Fig. 8 As proposed in [33], a custom orthogonal wavelet was designed for the IR images of the IC: this wavelet was derived from the Haar wavelet. Then, in our study, we use the Legendre complete basis for a multi-scale analysis of IR images. The polynomial degree and the grid's size for the sampling are chosen to fit our needs. In this previous study [33], different wavelets were compared in their ability to catch details of the image signal. The high-frequency wavelets catch finer details than the lower-frequency wavelets. The experiments show that the low-frequency wavelets (i.e. Haar and the custom one) are more adapted to analyze signal of the IR images. In that respect, as the frequency of the Legendrian polynomial increase the its degree, a low should be of interest. The decomposition of an image using a polynomial basis of degree 3 is presented in Fig. 9, where we find again the directional polynomial coefficients just as the wavelet coefficients. Depending on the binomial's degree (i, j) of the polynomials used during the image decomposition,
<table><caption>TABLE 1
Few Weighting Functions ω Used for Different Families of Polynomials</caption><thead><tr><td>Polynomial</td><td>Interval</td><td>ω</td></tr></thead><tbody><tr><td>Legendre</td><td>[-1; 1]<sup>2</sup></td><td>1</td></tr><tr><td>Hermite</td><td>[-∞; +∞]<sup>2</sup></td><td>e<sup>-<sup>x<sub>1</sub><sup>2</sup> + x<sub>2</sub><sup>2</sup></sup> / 2</sup></td></tr><tr><td>Laguerre</td><td>[0; +∞]<sup>2</sup></td><td>e<sup>-<sup>(x<sub>1</sub> + x<sub>2</sub>)</sup> / 2</sup></td></tr><tr><td>Tchebychev1</td><td>[-1; 1]<sup>2</sup></td><td>1 / √<span style="text-decoration: overline;">(1 - x<sub>1</sub>)<sup>2</sup>(1 - x<sub>2</sub>)<sup>2</sup></span></td></tr><tr><td>Tchebychev2</td><td>[-1; 1]<sup>2</sup></td><td>√<span style="text-decoration: overline;">(1 - x<sub>1</sub>)<sup>2</sup>(1 - x<sub>2</sub>)<sup>2</sup></span></td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 8. Two decompositions of an IR image (50x): 2D DWT using the Haar wavelet (left) and polynomial decomposition using the Legendrian basis with a 2 × 2 support.</figcaption></figure>

where $a_{d_1, d_2}$ is the polynomial coefficients

$$a_{d_1, d_2}(I) = \frac{\langle I | P_{d_1, d_2} \rangle_{\omega}}{\langle P_{d_1, d_2} | P_{d_1, d_2} \rangle_{\omega}} \quad (5)$$

This scalar product can be calculated as an integral, sampled on a 2D grid whose points (or collocation points) may be equally spaced (discrete uniform distribution). The size of this grid will determine the approximation's fineness at each location in the image. An algorithm is proposed in [44] to approximate this projection.

Similarly to the wavelet decomposition approach, the choice of the approximation function is of utmost importance. As discussed in Section 3.2, our images have some specifics (noise, texture, horizontal/vertical components, etc.) and we take care to extract specific data from these images. Thus, the purpose is to choose the best function to decompose the complex signal representing the data of our images. In [47], the similarity is put forward between the Legendre polynomial discretized on a reduced range (2x2 approximation grid) and the Haar wavelet. This similitude is illustrated in Fig. 8 As proposed in [33], a custom orthogonal wavelet was designed for the IR images of the IC: this wavelet was derived from the Haar wavelet. Then, in our study, we use the Legendre complete basis for a multi-scale analysis of IR images. The polynomial degree and the grid's size for the sampling are chosen to fit our needs. In this previous study [33], different wavelets were compared in their ability to catch details of the image signal. The high-frequency wavelets catch finer details than the lower-frequency wavelets. The experiments show that the low-frequency wavelets (i.e. Haar and the custom one) are more adapted to analyze signal of the IR images. In that respect, as the frequency of the Legendrian polynomial increase the its degree, a low should be of interest. The decomposition of an image using a polynomial basis of degree 3 is presented in Fig. 9, where we find again the directional polynomial coefficients just as the wavelet coefficients. Depending on the binomial's degree (i, j) of the polynomials used during the image decomposition,
<figure><img src="image_12.png" /><figcaption>Fig. 9. Polynomial coefficients (absolute values) of an IR image (50x) using a 3 × 3 Legendrian complete basis.</figcaption></figure>

different directional information are raised: the horizontal, vertical and diagonal components of the decomposition are respectively obtained for $(i, j) \in \{0, [0, d_1]\}, (i, j) \in \{[0, d_1], 0\}$ and $i = j$.

## 4.2 Polynomial-Based FM

Here we consider the following algorithm template, whose details are presented further below.

### 4.2.1 The Algorithm Template:

1) 2D PD of gray-scale image using a Legendre orthonormal basis, of degree $D$. Let $PC_{ij}$ denote the projection coefficients resulting from the projection of an image on a polynomial whose the two monomials are of degree $(i, j)$, where $i \in [0, d_1], j \in [0, d_2]$. The polynomial basis degree $D = \sup(d_1, d_2)$.

2) Given a statistic function $F$, $F_{PC_{ij}}$ means $F$ computed on $PC_{ij}$.

3) For each level $n \in [0, D]$, $F_n$ is defined as the pounding sum of each $F_{PC_{ij}, (i,j) \in \{[0,n] \cup [0,n]\}}$ as follows:

$$F_n = \alpha * F_{PC_{0,n}} + \beta * F_{PC_{n,0}} + \gamma * F_{PC_{n,n}} \quad (6)$$

where $\alpha$, $\beta$ and $\gamma$ represent respectively the weight of vertical, horizontal and diagonal decomposition components, and $\alpha + \beta + \gamma = 1$.

4) The total $F_{tot}$ of the PD is the sum of each $F_n$ weighted as follows:

$$F_{tot} = \sum_{n=0}^{D} 2^n F_n \quad (7)$$

### 4.2.2 Some Specifics Bound to the Previous Algorithm:

* As introduced in Section 3.2, we only consider the vertical and horizontal features of our images. Then we define the weights $\alpha = 0.5$, $\beta = 0.5$ and $\gamma = 0$. In other words, we only perform vertical and horizontal approximations.
<figure><img src="image_12.png" /><figcaption>Fig. 9. Polynomial coefficients (absolute values) of an IR image (50x) using a 3 × 3 Legendrian complete basis.</figcaption></figure>

different directional information are raised: the horizontal, vertical and diagonal components of the decomposition are respectively obtained for $(i, j) \in \{0, [0, d_1]\}, (i, j) \in \{[0, d_1], 0\}$ and $i = j$.

## 4.2 Polynomial-Based FM

Here we consider the following algorithm template, whose details are presented further below.

### 4.2.1 The Algorithm Template:

1) 2D PD of gray-scale image using a Legendre orthonormal basis, of degree $D$. Let $PC_{ij}$ denote the projection coefficients resulting from the projection of an image on a polynomial whose the two monomials are of degree $(i, j)$, where $i \in [0, d_1], j \in [0, d_2]$. The polynomial basis degree $D = \sup(d_1, d_2)$.

2) Given a statistic function $F$, $F_{PC_{ij}}$ means $F$ computed on $PC_{ij}$.

3) For each level $n \in [0, D]$, $F_n$ is defined as the pounding sum of each $F_{PC_{ij}, (i,j) \in \{[0,n] \cup [0,n]\}}$ as follows:

$$F_n = \alpha * F_{PC_{0,n}} + \beta * F_{PC_{n,0}} + \gamma * F_{PC_{n,n}} \quad (6)$$

where $\alpha$, $\beta$ and $\gamma$ represent respectively the weight of vertical, horizontal and diagonal decomposition components, and $\alpha + \beta + \gamma = 1$.

4) The total $F_{tot}$ of the PD is the sum of each $F_n$ weighted as follows:

$$F_{tot} = \sum_{n=0}^{D} 2^n F_n \quad (7)$$

### 4.2.2 Some Specifics Bound to the Previous Algorithm:

* As introduced in Section 3.2, we only consider the vertical and horizontal features of our images. Then we define the weights $\alpha = 0.5$, $\beta = 0.5$ and $\gamma = 0$. In other words, we only perform vertical and horizontal approximations.
* As stated in the previous section (Sec. 4.1), the polynomial basis used for our IR images is a Legendre complete basis. Moreover, we have also specified that polynomials of low degree (i.e. frequency) are of interest. To find which degree to use, we made several experiments of the FMPOD using a polynomial basis of degree $\in [2, 7]$, for acquisitions performed with different zoom factors. It is stated that the metric is more stable for the orthonormal polynomial basis of degree $d_1 = d_2 = D = 3$. Additionally, in order to obtain a good compromise between quality and computational time, we consider polynomials where $i \in \{0, D\}$ and $j \in \{0, D\}$, where the most the relevant information are held. It implies in this case:

$$F_{tot} = F_3 \qquad (8)$$

* The statistic function $F$ is defined as the SD as it reliably informs us about the data diversity. Then we consider the following assignment:

$$F \leftarrow \text{Standard Deviation} \qquad (9)$$

* Regarding the noise present in our images, a $3 \times 3$ sampling grid is selected for an improved approximation during the PD.

Finally, applying all these points, the $F_{tot}$ is our FMPOD. Results of focus research using the FMPOD are presented in the next section.

## 5. Experiments and Results

First, a video is acquired while moving the optical system in a linear movement towards the object of interest until it is as close as possible (physical limits)This is performed for each magnifying lens ($2.5x, 5x, 20x$ and $50x$) at a rate of 80 fps. Second, the images are extracted from the video, so that each image corresponds to a lens position, and then they are evaluated with a quality metric. In this way, the highest-ranked image should have the best matching focus according to the quality criterion. Four focus metrics are compared here:

1) The approach based on WPT from [31].

2) The Fast Index SHarPNess (FISH)-based approach presented in [32].

3) The COW-based method proposed in [33].

4) The polynomial-based approach using a Legendrian complete basis presented in the previous section: the FMPOD.

The focus measures (1), (2), (3) and (4) are presented in Fig. 10, for video acquisitions using $2.5x, 5x, 20x$ and $50x$ magnifying lenses, respectively. Fig. 11 reports the focused views relative to these acquisitions.

First, the FM 1 seems unable to find the correct focusing in most cases. As highlighted in Section 3.2, the highest frequencies in our images may be impacted by the thermal noise and the textural grain, which increase with magnification. It is a valuable explanation of the failure of this FM. The metric (3) takes its strength from the low frequencies preferred by the DWT combined with the COW. The COW is less sensitive to high frequencies than other wavelets. In this way, the AF algorithm is well suited by this FM and provides correct results for each zoom factor. However, some light distortion may lead to an unexpected behavior of the wavelet-based method: it corresponds to the left part of the blue curve in Figs. 10.b, c and d. The FMPOD (4) presents a strong precision and is not disturbed by any light distortions. In Fig. 10.a the variation visible around the frame number 880 may be caused by the packaging around the IC visible in Fig. 11.a. Notice that the contaminants, visible in Fig. 11.b, do not disturb the different FMs (Fig. 10.b).

In the case of an SOI with low contrast, the focused view is more difficult to be obtained. Such a case is presented in Fig. 12 with a video acquisition made using the $50x$ magnifying lens. Several light distortions are shown in Fig. 13.

One way to limit such light distortion around the focused position of the optical system is to manually configure the camera's WDR system to clamp the contrast optimization around a given value. This potentially needs to be reset when changing the lens and/or the IC, and will result in a lack of time.
* As stated in the previous section (Sec. 4.1), the polynomial basis used for our IR images is a Legendre complete basis. Moreover, we have also specified that polynomials of low degree (i.e. frequency) are of interest. To find which degree to use, we made several experiments of the FMPOD using a polynomial basis of degree $\in [2, 7]$, for acquisitions performed with different zoom factors. It is stated that the metric is more stable for the orthonormal polynomial basis of degree $d_1 = d_2 = D = 3$. Additionally, in order to obtain a good compromise between quality and computational time, we consider polynomials where $i \in \{0, D\}$ and $j \in \{0, D\}$, where the most the relevant information are held. It implies in this case:

$$F_{tot} = F_3 \qquad (8)$$

* The statistic function $F$ is defined as the SD as it reliably informs us about the data diversity. Then we consider the following assignment:

$$F \leftarrow \text{Standard Deviation} \qquad (9)$$

* Regarding the noise present in our images, a $3 \times 3$ sampling grid is selected for an improved approximation during the PD.

Finally, applying all these points, the $F_{tot}$ is our FMPOD. Results of focus research using the FMPOD are presented in the next section.

## 5. Experiments and Results

First, a video is acquired while moving the optical system in a linear movement towards the object of interest until it is as close as possible (physical limits). This is performed for each magnifying lens ($2.5x, 5x, 20x$ and $50x$) at a rate of 80 fps. Second, the images are extracted from the video, so that each image corresponds to a lens position, and then they are evaluated with a quality metric. In this way, the highest-ranked image should have the best matching focus according to the quality criterion. Four focus metrics are compared here:

1) The approach based on WPT from [31].

2) The Fast Index SHarPNess (FISH)-based approach presented in [32].

3) The COW-based method proposed in [33].

4) The polynomial-based approach using a Legendrian complete basis presented in the previous section: the FMPOD.

The focus measures (1), (2), (3) and (4) are presented in Fig. 10, for video acquisitions using $2.5x, 5x, 20x$ and $50x$ magnifying lenses, respectively. Fig. 11 reports the focused views relative to these acquisitions.

First, the FM 1 seems unable to find the correct focusing in most cases. As highlighted in Section 3.2, the highest frequencies in our images may be impacted by the thermal noise and the textural grain, which increase with magnification. It is a valuable explanation of the failure of this FM. The metric (3) takes its strength from the low frequencies preferred by the DWT combined with the COW. The COW is less sensitive to high frequencies than other wavelets. In this way, the AF algorithm is well suited by this FM and provides correct results for each zoom factor. However, some light distortion may lead to an unexpected behavior of the wavelet-based method: it corresponds to the left part of the blue curve in Figs. 10.b, c and d. The FMPOD (4) presents a strong precision and is not disturbed by any light distortions. In Fig. 10.a the variation visible around the frame number 880 may be caused by the packaging around the IC visible in Fig. 11.a. Notice that the contaminants, visible in Fig. 11.b, do not disturb the different FMs (Fig. 10.b).

In the case of an SOI with low contrast, the focused view is more difficult to be obtained. Such a case is presented in Fig. 12 with a video acquisition made using the $50x$ magnifying lens. Several light distortions are shown in Fig. 13.

One way to limit such light distortion around the focused position of the optical system is to manually configure the camera's WDR system to clamp the contrast optimization around a given value. This potentially needs to be reset when changing the lens and/or the IC, and will result in a lack of time.
<figure><img src="image_13.png" /><figcaption>Fig. 10. Focus metric in function of the distance optical-object: the abscesses is the index of an image in the video acquisition, the ordinate is the focus measure depending on the approach for each zoom factor (2.5x (a), 5x (b), 20x (c) and 50x (d)). FISH-based method in blue, WPT-based in green, COW in black and FMPOD in red). The right column shows a zoomed version of each curve around the focus position. The black dashed line represent the interval of focus, which can contain only one frame in 20x and 50x cases.</figcaption></figure>

To evaluate the robustness of our method, we applied additional distortions to the acquisitions: noise, blur and sharpness increase. These distortions may simulate a practical use case like using another optical system, a better or defective IR camera, a SOI with sharper material reflectivity, lower brightness, etc. Below are two synthetic tables of the focus success rate using each of the four methods (1, 2, 3 and 4) with respect to the distortion applied. We make the evaluation on two different datasets: in Table 14 the acquisition is done for a classical use case, i.e. for an SOI with normal contrast; in Table 15 the SOI is of poor quality, implying an SOI with lower contrast.
<figure><img src="image_13.png" /><figcaption>Fig. 10. Focus metric in function of the distance optical-object: the abscesses is the index of an image in the video acquisition, the ordinate is the focus measure depending on the approach for each zoom factor (2.5x (a), 5x (b), 20x (c) and 50x (d)). FISH-based method in blue, WPT-based in green, COW in black and FMPOD in red). The right column shows a zoomed version of each curve around the focus position. The black dashed line represent the interval of focus, which can contain only one frame in 20x and 50x cases.</figcaption></figure>

To evaluate the robustness of our method, we applied additional distortions to the acquisitions: noise, blur and sharpness increase. These distortions may simulate a practical use case like using another optical system, a better or defective IR camera, a SOI with sharper material reflectivity, lower brightness, etc. Below are two synthetic tables of the focus success rate using each of the four methods (1, 2, 3 and 4) with respect to the distortion applied. We make the evaluation on two different datasets: in Table 14 the acquisition is done for a classical use case, i.e. for an SOI with normal contrast; in Table 15 the SOI is of poor quality, implying an SOI with lower contrast.
<figure><img src="image_14.png" /><figcaption>Fig. 11. Focused view of SOI using different magnifying lenses: 2.5x (a), 5x (b), 20x (c) and 50x (d).</figcaption></figure>

**Standard case of study (Table 14):**

* The methods (1) and (2) give very poor results, albeit the rate of success (1) is more stable regardless the distortion than (2).

* Despite its robustness against additional noises and blurs, the DWT-based method (3) is sensitive to image sharpening, which may increase the ambient light distortions, and sensitive to Gaussian noise which may increase the thermal noise. This weakness may be highlighted for our new method (4) too. Nevertheless, the method (4) presents a high robustness to every type of distortion compared to the other FM tested.

**Limit case of study (Table 15):**

For the extreme case of study, the focus metrics reported in Table 15 are globally much worse than in the previous case; but the success rate of our method (4) is the best overall compared to Methods 1, 2 and 3.

These two experiments confirmed the robustness of our new method against the three others selected here.

**6. Implementation**

The FISH-, WPT-, COW-based methods and the FMPOD were implemented in Python language with additional features such as:

* PyWavelets package for the WPT and the DWT-based methods.
<figure><img src="image_14.png" /><figcaption>Fig. 11. Focused view of SOI using different magnifying lenses: 2.5x (a), 5x (b), 20x (c) and 50x (d).</figcaption></figure>

**Standard case of study (Table 14):**

* The methods (1) and (2) give very poor results, albeit the rate of success (1) is more stable regardless the distortion than (2).

* Despite its robustness against additional noises and blurs, the DWT-based method (3) is sensitive to image sharpening, which may increase the ambient light distortions, and sensitive to Gaussian noise which may increase the thermal noise. This weakness may be highlighted for our new method (4) too. Nevertheless, the method (4) presents a high robustness to every type of distortion compared to the other FM tested.

**Limit case of study (Table 15):**

For the extreme case of study, the focus metrics reported in Table 15 are globally much worse than in the previous case; but the success rate of our method (4) is the best overall compared to Methods 1, 2 and 3.

These two experiments confirmed the robustness of our new method against the three others selected here.

**6. Implementation**

The FISH-, WPT-, COW-based methods and the FMPOD were implemented in Python language with additional features such as:

* PyWavelets package for the WPT and the DWT-based methods.
<figure><img src="image_15.png" /><figcaption>Fig. 12. (a) Focus measure depending the approach: the FISH-based in blue, WPT-based in green, COW-based in black and FMPOD in red; (b) is the curve zoomed in around the focus position; (c) the focused view, correctly raised by the black and red curves.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 13. Examples of two phenomena for the light distorted by the working environment obtained with the 5x (a) and the 20x magnifying lens (b).</figcaption></figure>

* For the FMPOD (4), an additional tool was developed in C++ for the Polynomial decomposition, and made compatible with Python using PyBind11. Moreover, OpenMP was used for C++ loop parallelization on the CPU side.

A comparison of the different algorithms in terms of complexity and runtime may be irrelevant, since each implementation relies on very different technologies. Nevertheless, here is the runtime of each algorithm:

* FMPOD: 3.5 ms per image

* FM using COW: 2.24 ms per image

* FM using WPT: 3.7 ms per image
<figure><img src="image_15.png" /><figcaption>Fig. 12. (a) Focus measure depending the approach: the FISH-based in blue, WPT-based in green, COW-based in black and FMPOD in red; (b) is the curve zoomed in around the focus position; (c) the focused view, correctly raised by the black and red curves.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Fig. 13. Examples of two phenomena for the light distorted by the working environment obtained with the 5x (a) and the 20x magnifying lens (b).</figcaption></figure>

* For the FMPOD (4), an additional tool was developed in C++ for the Polynomial decomposition, and made compatible with Python using PyBind11. Moreover, OpenMP was used for C++ loop parallelization on the CPU side.

A comparison of the different algorithms in terms of complexity and runtime may be irrelevant, since each implementation relies on very different technologies. Nevertheless, here is the runtime of each algorithm:

* FMPOD: 3.5 ms per image

* FM using COW: 2.24 ms per image

* FM using WPT: 3.7 ms per image
<figure><img src="image_17.png" /><figcaption>Fig. 14. Rate of focus correctness depending on the FM and the distortion applied (colors) on images acquired in the standard use case (normal brightness and SOI contrasts).</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 15. Rate of focus correctness depending on the FM and the distortion applied (colors) on images acquired in the case of abnormal brightness and low contrasted SOI (limit case).</figcaption></figure>
<figure><img src="image_17.png" /><figcaption>Fig. 14. Rate of focus correctness depending on the FM and the distortion applied (colors) on images acquired in the standard use case (normal brightness and SOI contrasts).</figcaption></figure>

<figure><img src="image_18.png" /><figcaption>Fig. 15. Rate of focus correctness depending on the FM and the distortion applied (colors) on images acquired in the case of abnormal brightness and low contrasted SOI (limit case).</figcaption></figure>
* FM using FISH: 2.1 ms per image

In our application, the precision of the metric is more important than the runtime as long as the gap is only around 1ms. The runtime of FMPOD still matches our requirements in the context of the characterization of integrated circuits.

## 7. Conclusion

This article addresses the need for a focus criterion adapted to the particular field of IC microscopy. The infrared technology used in our study makes the experiment unique. Of all the state-of-the-art methods to evaluate the quality of an image, a few of them are adapted to the AF algorithms. This is even more striking for our context, whose specificities are multiple, such as the low level of contrasts, the noise and the texture. The polynomial-based method proposed in this paper (the FMPOD) takes into account all difficulties to develop a reliable focus measurement technique without additional image processing. Regardless of the optical magnifying lens, this FM determines the correct focal position on the surface of interest, through the silicon and inside the microscopic IC. Several measurements were made to confirm the robustness of our method against additional distortions, and compared to three other focusing methods. Finally, the precision and the time efficiency of the proposed algorithm make it a perfect candidate for the FM in the context of IC characterization.

A polynomial transform permits a local reconstruction of an image. By this way, we could build local feature descriptors to characterize regions of interest in an image, and use them in a pattern recognition process.

## Acknowledgment

The authors wish to thank the anonymous reviewers for their valuable suggestions.

## References

[1] B. Neumann, "Autofokussierung," *Leitz-Mitt. Wiss. Techn.*, vol. 8, no. 8, pp. 228-232, 1985.

[2] B. Neumann, A. Dämon, D. Hogenkamp, E. Beckmann, and J. Kollmann, "A laser-autofocus for automatic microscopy and metrology," *Sensors Actuators*, vol. 17, no. 1-2, pp. 267-272, 1989.

[3] M. Hansard, S. Lee, O. Choi, and R. Horaud, *Time of Flight Cameras: Principles, Methods, and Applications*. London: Springer, 2012. [Online]. Available: https://hal.inria.fr/hal-00725654

[4] S. Su, F. Heide, G. Wetzstein, and W. Heidrich, "Deep end-to-end time-of-flight imaging," in *Proc. IEEE Conf. Comput. Vision Pattern Recognit*. IEEE, 2018, pp. 6383-6392.

[5] S. Kurtti, J. Nissinen, J. Jansson, and J. Kostamovaara, "A cmos chip set for accurate pulsed time-of-flight laser range finding," in *Proc. IEEE Int. Instrum. Meas. Technol. Conf. (I2MTC)*. IEEE, 2018, pp. 1-5.

[6] A. Gershun, "The light field," *J. Math. Phys.*, vol. 18, no. 1-4, pp. 51-151, 1939. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/sapm193918151

[7] N. L. Stauffer, "Digital auto focus system utilizing a photodetector array," Mar. 2 1982, uS Patent 4,317,991.

[8] R. Ng et al., "Light field photography with a hand-held plenoptic camera," Stanford Univ. Computer Sci., Stanford, CA, USA, Tech. Rep. CSTR 2005-02, Apr. 2005.

[9] M. Levoy, "Light fields and computational imaging," *Computers*, vol. 39, no. 8, pp. 46-55, Aug. 2006.

[10] C. Zhang, G. Hou, Z. Zhang, Z. Sun, and T. Tan, "Efficient auto-refocusing for light field camera," *Pattern Recognit.*, vol. 81, pp. 176-189, 2018. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320318301110

[11] M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, "Light field microscopy," *ACM Trans. Graph. (TOG)*, vol. 25, no. 3, pp. 924-934, 2006.

[12] T. Iwane, "Light-field optics and its potential applications," in *Proc. 17th Workshop Inf. Opt. (WIO)*, Jul. 2018, pp. 1-3.

[13] M. Bathe-Peters, P. Annibale, and M. J. Lohse, "All-optical microscope autofocus based on an electrically tunable lens and a totally internally reflected ir laser," *Opt. Exp.*, vol. 26, no. 3, pp. 2359-2368, Feb. 2018. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-26-3-2359

[14] Z. Wang, A. C. Bovik, and L. Lu, "Why is image quality assessment so difficult?" in *Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.*, Orlando, FL, USA, May 2002, pp. IV-3313-IV-3316.

[15] H. Sheikh and A. Bovik, "Image information and visual quality," *IEEE Trans. Image Process.*, vol. 15, no. 2, pp. 430-444, Feb. 2006.

[16] V. Kamble and K. M. Bhurchandi, "No-reference image quality assessment algorithms: A survey," *Optik - Int. J. Light Electron Opt.*, vol. 126, no. 11-12, pp. 1090-1097, Jun. 2015.
* FM using FISH: 2.1 ms per image

In our application, the precision of the metric is more important than the runtime as long as the gap is only around 1ms. The runtime of FMPOD still matches our requirements in the context of the characterization of integrated circuits.

## 7. Conclusion

This article addresses the need for a focus criterion adapted to the particular field of IC microscopy. The infrared technology used in our study makes the experiment unique. Of all the state-of-the-art methods to evaluate the quality of an image, a few of them are adapted to the AF algorithms. This is even more striking for our context, whose specificities are multiple, such as the low level of contrasts, the noise and the texture. The polynomial-based method proposed in this paper (the FMPOD) takes into account all difficulties to develop a reliable focus measurement technique without additional image processing. Regardless of the optical magnifying lens, this FM determines the correct focal position on the surface of interest, through the silicon and inside the microscopic IC. Several measurements were made to confirm the robustness of our method against additional distortions, and compared to three other focusing methods. Finally, the precision and the time efficiency of the proposed algorithm make it a perfect candidate for the FM in the context of IC characterization.

A polynomial transform permits a local reconstruction of an image. By this way, we could build local feature descriptors to characterize regions of interest in an image, and use them in a pattern recognition process.

## Acknowledgment

The authors wish to thank the anonymous reviewers for their valuable suggestions.

## References

[1] B. Neumann, "Autofokussierung," *Leitz-Mitt. Wiss. Techn.*, vol. 8, no. 8, pp. 228-232, 1985.

[2] B. Neumann, A. Dämon, D. Hogenkamp, E. Beckmann, and J. Kollmann, "A laser-autofocus for automatic microscopy and metrology," *Sensors Actuators*, vol. 17, no. 1-2, pp. 267-272, 1989.

[3] M. Hansard, S. Lee, O. Choi, and R. Horaud, *Time of Flight Cameras: Principles, Methods, and Applications*. London: Springer, 2012. [Online]. Available: https://hal.inria.fr/hal-00725654

[4] S. Su, F. Heide, G. Wetzstein, and W. Heidrich, "Deep end-to-end time-of-flight imaging," in *Proc. IEEE Conf. Comput. Vision Pattern Recognit*. IEEE, 2018, pp. 6383-6392.

[5] S. Kurtti, J. Nissinen, J. Jansson, and J. Kostamovaara, "A cmos chip set for accurate pulsed time-of-flight laser range finding," in *Proc. IEEE Int. Instrum. Meas. Technol. Conf. (I2MTC)*. IEEE, 2018, pp. 1-5.

[6] A. Gershun, "The light field," *J. Math. Phys.*, vol. 18, no. 1-4, pp. 51-151, 1939. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/sapm193918151

[7] N. L. Stauffer, "Digital auto focus system utilizing a photodetector array," Mar. 2 1982, uS Patent 4,317,991.

[8] R. Ng et al., "Light field photography with a hand-held plenoptic camera," Stanford Univ. Computer Sci., Stanford, CA, USA, Tech. Rep. CSTR 2005-02, Apr. 2005.

[9] M. Levoy, "Light fields and computational imaging," *Computers*, vol. 39, no. 8, pp. 46-55, Aug. 2006.

[10] C. Zhang, G. Hou, Z. Zhang, Z. Sun, and T. Tan, "Efficient auto-refocusing for light field camera," *Pattern Recognit.*, vol. 81, pp. 176-189, 2018. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0031320318301110

[11] M. Levoy, R. Ng, A. Adams, M. Footer, and M. Horowitz, "Light field microscopy," *ACM Trans. Graph. (TOG)*, vol. 25, no. 3, pp. 924-934, 2006.

[12] T. Iwane, "Light-field optics and its potential applications," in *Proc. 17th Workshop Inf. Opt. (WIO)*, Jul. 2018, pp. 1-3.

[13] M. Bathe-Peters, P. Annibale, and M. J. Lohse, "All-optical microscope autofocus based on an electrically tunable lens and a totally internally reflected ir laser," *Opt. Exp.*, vol. 26, no. 3, pp. 2359-2368, Feb. 2018. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-26-3-2359

[14] Z. Wang, A. C. Bovik, and L. Lu, "Why is image quality assessment so difficult?" in *Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.*, Orlando, FL, USA, May 2002, pp. IV-3313-IV-3316.

[15] H. Sheikh and A. Bovik, "Image information and visual quality," *IEEE Trans. Image Process.*, vol. 15, no. 2, pp. 430-444, Feb. 2006.

[16] V. Kamble and K. M. Bhurchandi, "No-reference image quality assessment algorithms: A survey," *Optik - Int. J. Light Electron Opt.*, vol. 126, no. 11-12, pp. 1090-1097, Jun. 2015.
[17] H. Kim, S. Kang, H. Kang, Y. Cho, N. Park, and J. Kim, "Automatic focus control for assembly alignment in a lens module process," in *Proc. IEEE Int. Symp. Assem. and Manuf.*, IEEE, Nov. 2009, pp. 292–297.

[18] R. C. Streijl, S. Winkler, and D. S. Hands, "Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives," *Multimedia Syst.*, vol. 22, no. 2, pp. 213–227, Mar. 2016.

[19] J. F. Schlag, A. C. Sanderson, C. P. Neuman, and F. C. Wimberly, "Implementation of automatic focusing algorithms for a computer vision system with camera control," Robotics Institute, Carnegie-Mellon University, Tech. Rep., 1983.

[20] E. Krotkov, "Focusing," *Int. J. Comput. Vision*, vol. 1, no. 3, pp. 223–237, Oct. 1988.

[21] X. Xu, Y. Wang, J. Tang, X. Zhang, and X. Liu, "Robust automatic focus algorithm for low contrast images using a new contrast measure," *Sensors*, vol. 11, no. 12, pp. 8281–8294, Aug. 2011.

[22] Y. Bian, Y. Zhang, P. Yin, H. Li, and A. Ozcan, "Optical refractometry using lensless holography and autofocusing," *Opt. Exp.*, vol. 26, no. 23, pp. 29 614–29 628, 2018.

[23] Y. Zhang, H. Wang, Y. Wu, M. Tamamitsu, and A. Ozean, "Robust holographic autofocusing based on edge sparsity," in *Proc. Conf. Lasers Electro-Opt. (CLEO)*, San Jose, CA, USA, May 2018, pp. 1–2.

[24] M. Tamamitsu, Y. Zhang, H. Wang, Y. Wu, and A. Ozcan, "Comparison of gini index and tamura coefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront," 2017, *arXiv preprint arXiv:1708.08055*. [Online]. Available: https://arxiv.org/abs/1708.08055

[25] V. Kumar and P. Gupta, "Importance of statistical measures in digital image processing," *Int. J. Emerg. Technol. Adv. Eng.*, vol. 2, no. 8, pp. 56–62, Aug. 2012.

[26] X. Zhang, C. Jia, and K. Xie, "Evaluation of autofocus algorithm for automatic detection of caenorhabditis elegans lipid droplets," *Prog. Biochemistry Biophys.*, vol. 43, pp. 167–175, 2016.

[27] E. S. R. Fonseca, P. T. Fiadeiro, M. Pereira, and A. Pinheiro, "Comparative analysis of autofocus functions in digital in-line phase-shifting holography, autofocus," *Appl. Opt.*, vol. 55, no. 27, p. 7663, Sep. 2016.

[28] S. Podlech, "Autofocus by bayes spectral entropy applied to optical microscopy," *Microscopy Microanalysis*, vol. 22, no. 01, pp. 199–207, Jan. 2016.

[29] X. Zhang, H. Wu, and Y. Ma, "A new auto-focus measure based on medium frequency discrete cosine transform filtering and discrete cosine transform," *Appl. Comput. Harmon. Anal.*, vol. 40, no. 2, pp. 430–437, Mar. 2016.

[30] Z. Zhang, Y. Liu, Z. Xiong, J. Li, and M. Zhang, "Focus and blurriness measure using reorganized dct coefficients for an autofocus application," *IEEE Trans. Circuits Syst. Video Technol.*, vol. 28, no. 1, pp. 15–30, Jan. 2018.

[31] Z. Fan, S. Chen, H. Hu, H. Chang, and Q. Fu, "Autofocus algorithm based on wavelet packet transform for infrared microscopy," in *Proc. 3rd Int. Congr. Image Signal Process.*, vol. 5, Oct. 2010, pp. 2510–2514.

[32] P. V. Vu and D. M. Chandler, "A fast wavelet-based algorithm for global and local image sharpness estimation," *IEEE Signal Process. Lett.*, vol. 19, no. 7, pp. 423–426, Jul. 2012.

[33] R. Abele, D. Fronte, P. Y. Liardet, J. Boi, J. Damoiseaux, and D. Merad, "Autofocus in infrared microscopy," in *Proc. IEEE 23rd Int. Conf. Emerg. Technol. Factory Autom. (ETFA)*, vol. 1, Sep. 2018, pp. 631–637.

[34] J.-L. Starck and J. M. Fadili, "Numerical issues when using wavelets," in *Encyclopedia of Complexity and Systems Science*, R. A. Meyers, Ed. New York, NY, USA: Springer, 2009, pp. 6352–6368. [Online]. Available: https://doi.org/10.1007/978-0-387-30440-3_374

[35] M. A. Green and M. J. Keevers, "Optical properties of intrinsic silicon at 300 k," *Prog. Photovolt.: Res. Appl.*, vol. 3, no. 3, pp. 189–192, 1995.

[36] M. A. Green, "Self-consistent optical parameters of intrinsic silicon at 300 k including temperature coefficients," *Sol. Energy Mater. Sol. Cells*, vol. 92, no. 11, pp. 1305–1310, 2008.

[37] R. G. Stockwell, L. Mansinha, and R. P. Lowe, "Localization of the complex spectrum: The s transform," *IEEE Trans. Signal Process.*, vol. 44, no. 4, pp. 998–1001, Apr. 1996.

[38] A. Bhandari, P. Marziliano, and A. M. Barrutia, "Need for speed: Fast stockwell transform (fst) with o(n) complexity," in *Proc. 7th Int. Conf. Inf. Commun. Signal Process.*, Dec. 2009, pp. 1–5.

[39] M. Eden, M. Unser, and R. Leonardi, "Polynomial representation of pictures," *Signal Process.*, vol. 10, no. 4, pp. 385–393, 1986.

[40] I. Daubechies, *Ten Lectures on Wavelets*, vol. 61. Philadelphia: SIAM, 1992.

[41] J. Ma and G. Plonka, "The curvelet transform," *IEEE Signal Process. Mag.*, vol. 27, no. 2, pp. 118–133, Mar. 2010.

[42] N.-G. Cho, A. Yuille, and S.-W. Lee, "A novel linelet-based representation for line segment detection," *IEEE Trans. Pattern Anal. and Mach. Intell.*, vol. 40, no. 5, pp. 1195–1208, May 2017.

[43] O. Kihl, B. Tremblais, and B. Augereau, "Multivariate orthogonal polynomials to extract singular points," in *Proc. 15th IEEE Int. Conf. Image Process*. IEEE, 2008, pp. 857–860.

[44] C. Bordei, P. Bourdon, B. Augereau, and P. Carré, "Polynomial based texture representation for facial expression recognition, polynomial," in *Proc. IEEE Int. Conf. Acoust., Speech, Signal Process*. IEEE, May 2014, pp. 529–533.

[45] R. E. Moubtahij, B. Augereau, C. Fernandez-Maloigne, and H. Tairi, "A polynomial texture extraction with application in dynamic texture classification," in *Proc. 12th Int. Conf. Quality Control Artif. Vision*, vol. 9534. Bellingham, WA, USA: SPIE, Apr. 2015, p. 953407. [Online]. Available: http://adsabs.harvard.edu/abs/2015SPIE.9534E.07E

[46] A. S. Blaivas, "Visual analysis in unspecialized receptive fields as an orthogonal series expansion," *Neurophysiology*, vol. 6, no. 2, pp. 168–173, Mar. 1974. [Online]. Available: https://doi.org/10.1007/BF01062755

[47] R. E. Moubtahij, "Transformations polynomiales: Applications l'estimation de mouvements et la classification de vidos," Ph.D. thesis, Universit de Poitiers, 2016. [Online]. Available: http://these.univ-poitiers.fr

[48] O. Kihl, "Modlisations polynomiales hirarchises applications lanalyse de mouvements complexes," Ph.D. dissertation, Universit de Poitiers, 2012.
[17] H. Kim, S. Kang, H. Kang, Y. Cho, N. Park, and J. Kim, "Automatic focus control for assembly alignment in a lens module process," in *Proc. IEEE Int. Symp. Assem. and Manuf.*, IEEE, Nov. 2009, pp. 292-297.

[18] R. C. Streijl, S. Winkler, and D. S. Hands, "Mean opinion score (MOS) revisited: Methods and applications, limitations and alternatives," *Multimedia Syst.*, vol. 22, no. 2, pp. 213-227, Mar. 2016.

[19] J. F. Schlag, A. C. Sanderson, C. P. Neuman, and F. C. Wimberly, "Implementation of automatic focusing algorithms for a computer vision system with camera control," Robotics Institute, Carnegie-Mellon University, Tech. Rep., 1983.

[20] E. Krotkov, "Focusing," *Int. J. Comput. Vision*, vol. 1, no. 3, pp. 223-237, Oct. 1988.

[21] X. Xu, Y. Wang, J. Tang, X. Zhang, and X. Liu, "Robust automatic focus algorithm for low contrast images using a new contrast measure," *Sensors*, vol. 11, no. 12, pp. 8281-8294, Aug. 2011.

[22] Y. Bian, Y. Zhang, P. Yin, H. Li, and A. Ozcan, "Optical refractometry using lensless holography and autofocusing," *Opt. Exp.*, vol. 26, no. 23, pp. 29 614-629 628, 2018.

[23] Y. Zhang, H. Wang, Y. Wu, M. Tamamitsu, and A. Ozean, "Robust holographic autofocusing based on edge sparsity," in *Proc. Conf. Lasers Electro-Opt. (CLEO)*, San Jose, CA, USA, May 2018, pp. 1-2.

[24] M. Tamamitsu, Y. Zhang, H. Wang, Y. Wu, and A. Ozcan, "Comparison of gini index and tamura coefficient for holographic autofocusing based on the edge sparsity of the complex optical wavefront," 2017, *arXiv preprint arXiv:1708.08055*. [Online]. Available: https://arxiv.org/abs/1708.08055

[25] V. Kumar and P. Gupta, "Importance of statistical measures in digital image processing," *Int. J. Emerg. Technol. Adv. Eng.*, vol. 2, no. 8, pp. 56-62, Aug. 2012.

[26] X. Zhang, C. Jia, and K. Xie, "Evaluation of autofocus algorithm for automatic detection of caenorhabditis elegans lipid droplets," *Prog. Biochemistry Biophys.*, vol. 43, pp. 167-175, 2016.

[27] E. S. R. Fonseca, P. T. Fiadeiro, M. Pereira, and A. Pinheiro, "Comparative analysis of autofocus functions in digital in-line phase-shifting holography, autofocus," *Appl. Opt.*, vol. 55, no. 27, p. 7663, Sep. 2016.

[28] S. Podlech, "Autofocus by bayes spectral entropy applied to optical microscopy," *Microscopy Microanalysis*, vol. 22, no. 01, pp. 199-207, Jan. 2016.

[29] X. Zhang, H. Wu, and Y. Ma, "A new auto-focus measure based on medium frequency discrete cosine transform filtering and discrete cosine transform," *Appl. Comput. Harmon. Anal.*, vol. 40, no. 2, pp. 430-437, Mar. 2016.

[30] Z. Zhang, Y. Liu, Z. Xiong, J. Li, and M. Zhang, "Focus and blurriness measure using reorganized dct coefficients for an autofocus application," *IEEE Trans. Circuits Syst. Video Technol.*, vol. 28, no. 1, pp. 15-30, Jan. 2018.

[31] Z. Fan, S. Chen, H. Hu, H. Chang, and Q. Fu, "Autofocus algorithm based on wavelet packet transform for infrared microscopy," in *Proc. 3rd Int. Congr. Image Signal Process.*, vol. 5, Oct. 2010, pp. 2510-2514.

[32] P. V. Vu and D. M. Chandler, "A fast wavelet-based algorithm for global and local image sharpness estimation," *IEEE Signal Process. Lett.*, vol. 19, no. 7, pp. 423-426, Jul. 2012.

[33] R. Abele, D. Fronte, P. Y. Liardet, J. Boi, J. Damoiseaux, and D. Merad, "Autofocus in infrared microscopy," in *Proc. IEEE 23rd Int. Conf. Emerg. Technol. Factory Autom. (ETFA)*, vol. 1, Sep. 2018, pp. 631-637.

[34] J.-L. Starck and J. M. Fadili, "Numerical issues when using wavelets," in *Encyclopedia of Complexity and Systems Science*, R. A. Meyers, Ed. New York, NY, USA: Springer, 2009, pp. 6352-6368. [Online]. Available: https://doi.org/10.1007/978-0-387-30440-3_374

[35] M. A. Green and M. J. Keevers, "Optical properties of intrinsic silicon at 300 k," *Prog. Photovolt.: Res. Appl.*, vol. 3, no. 3, pp. 189-192, 1995.

[36] M. A. Green, "Self-consistent optical parameters of intrinsic silicon at 300 k including temperature coefficients," *Sol. Energy Mater. Sol. Cells*, vol. 92, no. 11, pp. 1305-1310, 2008.

[37] R. G. Stockwell, L. Mansinha, and R. P. Lowe, "Localization of the complex spectrum: The s transform," *IEEE Trans. Signal Process.*, vol. 44, no. 4, pp. 998-1001, Apr. 1996.

[38] A. Bhandari, P. Marziliano, and A. M. Barrutia, "Need for speed: Fast stockwell transform (fst) with o(n) complexity," in *Proc. 7th Int. Conf. Inf. Commun. Signal Process.*, Dec. 2009, pp. 1-5.

[39] M. Eden, M. Unser, and R. Leonardi, "Polynomial representation of pictures," *Signal Process.*, vol. 10, no. 4, pp. 385-393, 1986.

[40] I. Daubechies, *Ten Lectures on Wavelets*, vol. 61. Philadelphia: SIAM, 1992.

[41] J. Ma and G. Plonka, "The curvelet transform," *IEEE Signal Process. Mag.*, vol. 27, no. 2, pp. 118-133, Mar. 2010.

[42] N.-G. Cho, A. Yuille, and S.-W. Lee, "A novel linelet-based representation for line segment detection," *IEEE Trans. Pattern Anal. and Mach. Intell.*, vol. 40, no. 5, pp. 1195-1208, May 2017.

[43] O. Kihl, B. Tremblais, and B. Augereau, "Multivariate orthogonal polynomials to extract singular points," in *Proc. 15th IEEE Int. Conf. Image Process*. IEEE, 2008, pp. 857-860.

[44] C. Bordei, P. Bourdon, B. Augereau, and P. Carré, "Polynomial based texture representation for facial expression recognition, polynomial," in *Proc. IEEE Int. Conf. Acoust., Speech, Signal Process*. IEEE, May 2014, pp. 529-533.

[45] R. E. Moubtahij, B. Augereau, C. Fernandez-Maloigne, and H. Tairi, "A polynomial texture extraction with application in dynamic texture classification," in *Proc. 12th Int. Conf. Quality Control Artif. Vision*, vol. 9534. Bellingham, WA, USA: SPIE, Apr. 2015, p. 953407. [Online]. Available: http://adsabs.harvard.edu/abs/2015SPIE.9534E.07E

[46] A. S. Blaivas, "Visual analysis in unspecialized receptive fields as an orthogonal series expansion," *Neurophysiology*, vol. 6, no. 2, pp. 168-173, Mar. 1974. [Online]. Available: https://doi.org/10.1007/BF01062755

[47] R. E. Moubtahij, "Transformations polynomiales: Applications l'estimation de mouvements et la classification de vidos," Ph.D. thesis, Universit de Poitiers, 2016. [Online]. Available: http://these.univ-poitiers.fr

[48] O. Kihl, "Modlisations polynomiales hirarchises applications lanalyse de mouvements complexes," Ph.D. dissertation, Universit de Poitiers, 2012.