# Accurate 3D Surface Reconstruction for Smart Farming Application with an Inexpensive Shape from Focus System

Authors: Sung-An Lee; Byung-Geun Lee
Journal: Journal of Sensors
 (2020)

---

## Abstract

In precision agriculture, 3D vision systems are becoming increasingly important. By applying different optical 3D vision techniques, the acquired 3D data can provide information regarding the most important phenotype features in every agricultural scenario. However, most of these 3D vision systems are expensive, except some of the triangulation techniques. In this study, we focus on estimating accurate shapes using shape from focus (SFF), which is a triangulation technique. Typically, the SFF system incurs significant errors from images, including noise. As a solution to this problem, a simple low-pass filter such as the Gaussian filter has generally been used in most studies. However, when a low filter is applied, the noise is depressed but the signals are also blurred, which results in inaccuracies regarding the depth map. In this study, the noise is depressed independently without losing the original signals, and the edge components, which play important roles in finding a focused surface, are enhanced using the independent component analysis (ICA). The edge signals are amplified with a simple basis vector correction in the IC vector space. The experiments are implemented with simulated objects and real objects. The experimental results demonstrate that the obtained accuracy is comparable to that of existing methods.


---

## Body

## 1. Introduction

Precision agriculture requires sensing methodologies that provide information about individual crops and animals. During the last decade, 3D vision has become a key technology in precision agriculture for their extended capabilities compared to 2D. As 3D sensors are becoming smaller and smarter, the number of studies and applications related to agricultural 3D vision systems is increasing rapidly [1]. The 3D image generation techniques are mainly classified into three categories according to their principal measurement, namely, triangulation, time of flight, and interferometry. However, most of these 3D vision systems are of high cost owing to the expensive lasers or complicated scanning mirror systems involved, and hence, they are limited in their application, except some of triangulation techniques [2–4]. In the literature, many of the studies are aimed at improving the performance of triangulation methods in agricultural applications [4–7].The cost can be reduced by applying passive optical methods. Shape from focus (SFF) system, a passive optical method that uses the image focus, is less expensive than other 3D solutions because of the simplicity of the system configuration and provides high accuracy. The expensive optical elements are not needed, except only one CCD camera. The purpose of SFF is to estimate the shape of objects by finding the exact focused position of each pixel from the scene. In the system, an image sequence of the scene is obtained from a fixed point of view, changing the focus consecutively with a predefined step. Then, SFF infers the 3D shape of the object from the image sequence. Many of the SFF studies have sought to improve the 3D shape in multiple ways. The research on SFF techniques is mainly divided into three categories: focus measure (FM), approximation, and optimisation techniques. FM is defined as the local sharpness measure of each pixel on the image. The initial depth map is obtained after applying the FM to the image to find the maximum response on every pixel from the image. The sum of the modified Laplacian, Tenenbaum (TEN), and gray level variance are generally used in 3D shape recovery [8, 9]. After FM is applied, approximation techniques refine the initial depth map. Various focus curve fitting methods with planar and curved approximation for searching the exact focused image surface have been developed for better accuracy and speed. Different optimisation techniques have been applied, such as machine learning, and the hardware has been improved to correspond to the specific industrial field requirements and achieve better efficiency. Various algorithms have been used, such as neural network (SFF.NN), dynamic programming (SFF.DP), fuzzy logic (SFF.FL), and principal component analysis (SFF.PCA) [8–10].Compared to other imaging systems, SFF has some inherent errors due to translation, magnification, and its discrete number of frames. CCD noise is the main cause of error in the SFF system. Therefore, preprocessing to reduce noise is an essential part of most SFF algorithms. Convolving the original image with a Gaussian filter is the typical way to reduce the noise effect [11]. However, this method is not effective with a severely noisy dataset because important signals are also averaged. Thus, the noise effect still remains, which results in failure to find the accurate shape of objects.


## 2. Materials and Methods

The independent component analysis (ICA) is a useful tool for the separation of a set of signals from the original signal. The ICA decorrelates the mixed signaly assuming that the signal is composed of an independent signal vector s and a mixing matrix A. It can be expressed by the following equation:
(1)y=As.This equation can also be expressed as follows, with the assumption of a square matrix ofA:
(2)s=Wy.The goal of the ICA algorithm is to find the separation matrixW.


### 2.1. Preprocessing

(1)
Vector population: the input imageIxi,yj,zk, composed of Z frames, is converted by the vector vi,j,k for each pixel, where i=0,1,2,3⋯,X, j=0,1,2,3⋯,Y, and k=0,1,2,3⋯,Z, in each of the X×Y dimensions. The vector population is defined by Equation (3)
(3)vi,j,k=pi−1,j,kpi,j−1,kpi,j,k−1pi,j,kpi+1,j,kpi,+1,kpi,j,k+1T.Here,vi,j,k consists of seven neighbouring pixels. Figure 1 shows vi,j,k in the Cartesian coordinate.
(2)
Zero mean and whitening:vi,j,k is normalised considering the first- and second-order statistics. The equations are as follows:
(4)v=v−v,¯v′=Mv=D−1/2ETv.Figure 1
Vector population in the Cartesian coordinate.Here,v¯ is the mean of v and v′ is the covariance matrix, D is the eigenvalue matrix, and E is the orthogonal eigenvector.


### 2.2. FastICA

To divide the independent components from the original vector, the separation matrixW is calculated. In this study, a fixed-point FastICA is used for its generality. W is induced by minimisation of mutual information. The mutual information is defined using negentropy, which is the index of non-Gaussianity. The negentropy Jyis defined by the following equation:
(5)Jy=HyGauss−Hy.Here,Hyis the entropy for y, and yGauss is a Gaussian with unit variance. The minimisation of mutual information is equal to a maximisation of the negentropy. Here, the FastICA algorithm yields the solution as the following equation [12]:
(6)W+=ExgWTx−Eg′WTxWv′=Mv=D−1/2ETv,W∗=W+W+.Here,g is the first derivative of a nonquadratic nonlinear function, and E means the averaging over all column vectors of matrix x. The calculation of W∗ is repeated until convergence.


### 2.3. Denoising

Assuming that the original signal is contaminated by white Gaussian noise, the shrinkage function is used to find the noise components. Here, the probability distribution of the function is assumed as Laplacian [13]. The shrinkage function fs is followed by
(7)fs=11−σ2signsmax0,s−2σ2,where σ2 denotes the covariance of the Gaussian noise and the noise level of the image.


### 2.4. Edge Enhancement

Typically, the focus is on the surface of objects, that is, the edge components in the images. Thus, enhancing the edge component is one of the strategies to improve the depth map. From the independent signals, the edge components are enhanced by eliminating the components that maximise the mutual information. The component involves common information such as the average images between two consecutive frames. Here, the components are removed by simply replacing the value of the basis vector of W to zero.


### 2.5. Transition to Principal Components

Finally, the signal is changed to principal components from the independent components. After the transition, the final depth map is estimated using the first principal component, which has maximal covariance. Figure2 shows the transition stages of the proposed method. The final depth map Di,j is calculated by the following equation:
(8)Di,j=maxabsfk1.Figure 2
Transition stages of the proposed process.Here,fk1is the first principal component of the input signal.


## 2.1. Preprocessing

(1)
Vector population: the input imageIxi,yj,zk, composed of Z frames, is converted by the vector vi,j,k for each pixel, where i=0,1,2,3⋯,X, j=0,1,2,3⋯,Y, and k=0,1,2,3⋯,Z, in each of the X×Y dimensions. The vector population is defined by Equation (3)
(3)vi,j,k=pi−1,j,kpi,j−1,kpi,j,k−1pi,j,kpi+1,j,kpi,+1,kpi,j,k+1T.Here,vi,j,k consists of seven neighbouring pixels. Figure 1 shows vi,j,k in the Cartesian coordinate.
(2)
Zero mean and whitening:vi,j,k is normalised considering the first- and second-order statistics. The equations are as follows:
(4)v=v−v,¯v′=Mv=D−1/2ETv.Figure 1
Vector population in the Cartesian coordinate.Here,v¯ is the mean of v and v′ is the covariance matrix, D is the eigenvalue matrix, and E is the orthogonal eigenvector.


## 2.2. FastICA

To divide the independent components from the original vector, the separation matrixW is calculated. In this study, a fixed-point FastICA is used for its generality. W is induced by minimisation of mutual information. The mutual information is defined using negentropy, which is the index of non-Gaussianity. The negentropy Jyis defined by the following equation:
(5)Jy=HyGauss−Hy.Here,Hyis the entropy for y, and yGauss is a Gaussian with unit variance. The minimisation of mutual information is equal to a maximisation of the negentropy. Here, the FastICA algorithm yields the solution as the following equation [12]:
(6)W+=ExgWTx−Eg′WTxWv′=Mv=D−1/2ETv,W∗=W+W+.Here,g is the first derivative of a nonquadratic nonlinear function, and E means the averaging over all column vectors of matrix x. The calculation of W∗ is repeated until convergence.


## 2.3. Denoising

Assuming that the original signal is contaminated by white Gaussian noise, the shrinkage function is used to find the noise components. Here, the probability distribution of the function is assumed as Laplacian [13]. The shrinkage function fs is followed by
(7)fs=11−σ2signsmax0,s−2σ2,where σ2 denotes the covariance of the Gaussian noise and the noise level of the image.


## 2.4. Edge Enhancement

Typically, the focus is on the surface of objects, that is, the edge components in the images. Thus, enhancing the edge component is one of the strategies to improve the depth map. From the independent signals, the edge components are enhanced by eliminating the components that maximise the mutual information. The component involves common information such as the average images between two consecutive frames. Here, the components are removed by simply replacing the value of the basis vector of W to zero.


## 2.5. Transition to Principal Components

Finally, the signal is changed to principal components from the independent components. After the transition, the final depth map is estimated using the first principal component, which has maximal covariance. Figure2 shows the transition stages of the proposed method. The final depth map Di,j is calculated by the following equation:
(8)Di,j=maxabsfk1.Figure 2
Transition stages of the proposed process.Here,fk1is the first principal component of the input signal.


## 3. Results and Discussion

### 3.1. Experimental Setup

For the experiments, a simulated cone, which was created by a virtual program, is used. The simulated cone consists of 97 frames sized360×360. Figure 3 shows the real objects used for the experiments, which are a real cone object composed of 30 frames sized 200×200, a TFT-LCD cell composed of 60 frames sized 300×300, and the Lincoln head on a US penny composed of 60 frames sized 300×300. Figure 4 shows the simulated cone with 30 and 60 frames. It has a runtime of only 1 min with dual core Intel i3-2100 processors running at 3.1 GHz and 8 GB RAM in MATLAB.Figure 3
Real objects. (a) Real cone object and TFT-LCD cell. (b) Lincoln head on a US penny.
(a)
(b)Figure 4
The simulated cone. (a) 30 frames, (b) 60 frames.
(a)
(b)


### 3.2. Quantitative Analysis

For the quantitative analysis of the proposed algorithm, the root mean square error (RMSE) and correlation are calculated for each algorithm. The RMSE is an indicator of the amount of error in the images. It is denoted as(9)RMSE=1XY∑x=0X−1∑y=0Y−1fx,y−gx,y2,where X and Y are the number of pixels in the horizontal and vertical directions in the image, respectively, fx,y is the reference image, and gx,y is the resulting image of the algorithm. Correlation is the similarity of the two images. It is denoted as
(10)Correlation=∑m∑nFmn−F¯Gmn−G¯∑m∑nFmn−F¯2∑m∑nGmn−G¯2,where F and G are the reference image and the resulting image of the algorithm, respectively, and F¯ and G¯ are their means.In the case of experiments for synthetic object, it was possible to calculate RMSE and correlation as their true depth maps are known. For evaluating the performance of the proposed method, we experimented with three different Gaussian noise situations by changing the variance. The first situation is noise free, i.e.,σ2=0, the second isσ2=0.05, and the third is σ2=0.1. Figure 5 shows the simulated cone with additive Gaussian noise.Figure 5
The simulated cone with additive Gaussian noiseσ2=0.05 (a, b) and σ2=0.1 (c, d), with 30 frames (a, c) and 60 frames (b, d).
(a)
(b)
(c)
(d)The traditional SFF, which simply uses a Gaussian filter, the SFF.PCA, and the proposed method were applied to remove these noise effects, recover the 3D shape, and compare it with respect to the above two matrices. Table1 presents the robustness of the proposed algorithm compared with that of the others. In the table, it is clear that our approach significantly improved the results in comparison with the other methods. In a severe noise situation, when σ2=0.1, the RMSE is improved by 30% and 38% compared to the traditional SFF and SFF.PCA, and the correlation is improved by 99% and 139% compared to the traditional SFF and SFF.PCA, respectively.Table 1
Results of RMSE and correlation.
Method
Variance of Gaussian noise
0
0.05
0.1
Traditional SFF
RMSE
8.9320
15.607
35.384
Correlation
0.8629
0.6131
0.0875
SFF.PCA
RMSE
8.1961
8.4943
31.220
Correlation
0.9453
0.9073
0.1097
Proposed
RMSE
7.0132
8.0385
21.912
Correlation
0.9691
0.9435
0.2625


### 3.3. Qualitative Analysis

Figure6 shows the restored 3D shapes of the simulated cone. The first row is the traditional SFF, the second is SFF.PCA, and the third one is the proposed method. The first column is the noise-free situation, the second column isσ2=0.05, and the third one is σ2=0.1. In all the cases, it is shown that the results of the proposed algorithm have a smoother surface compared to that of others. Figure 7 shows the restored 3D shapes of the real objects with noise variance of 0.05. The first column is the traditional SFF, the second is SFF.PCA, and the third one is the proposed method. In all the cases, the proposed method excels in comparison with the others. It is shown that the results of the proposed algorithm have a smoother surface compared to that of the others.Figure 6
Restored 3D shape of the simulated cone. The variances are 0 (a, d, g), 0.05 (b, e, h), and 0.1 (c, f, i). Traditional method (a–c), SFF.PCA (d–f), and proposed method (g–i).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 7
Restored 3D shape of real objects with noise variance 0.05. Real cone object (a–c), TFT-LCD cell (d–f), and Lincoln head on US (g–i). Traditional method (a, d, g), SFF.PCA (b, e, h), and the proposed method (c, f, i).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)


## 3.1. Experimental Setup

For the experiments, a simulated cone, which was created by a virtual program, is used. The simulated cone consists of 97 frames sized360×360. Figure 3 shows the real objects used for the experiments, which are a real cone object composed of 30 frames sized 200×200, a TFT-LCD cell composed of 60 frames sized 300×300, and the Lincoln head on a US penny composed of 60 frames sized 300×300. Figure 4 shows the simulated cone with 30 and 60 frames. It has a runtime of only 1 min with dual core Intel i3-2100 processors running at 3.1 GHz and 8 GB RAM in MATLAB.Figure 3
Real objects. (a) Real cone object and TFT-LCD cell. (b) Lincoln head on a US penny.
(a)
(b)Figure 4
The simulated cone. (a) 30 frames, (b) 60 frames.
(a)
(b)


## 3.2. Quantitative Analysis

For the quantitative analysis of the proposed algorithm, the root mean square error (RMSE) and correlation are calculated for each algorithm. The RMSE is an indicator of the amount of error in the images. It is denoted as(9)RMSE=1XY∑x=0X−1∑y=0Y−1fx,y−gx,y2,where X and Y are the number of pixels in the horizontal and vertical directions in the image, respectively, fx,y is the reference image, and gx,y is the resulting image of the algorithm. Correlation is the similarity of the two images. It is denoted as
(10)Correlation=∑m∑nFmn−F¯Gmn−G¯∑m∑nFmn−F¯2∑m∑nGmn−G¯2,where F and G are the reference image and the resulting image of the algorithm, respectively, and F¯ and G¯ are their means.In the case of experiments for synthetic object, it was possible to calculate RMSE and correlation as their true depth maps are known. For evaluating the performance of the proposed method, we experimented with three different Gaussian noise situations by changing the variance. The first situation is noise free, i.e.,σ2=0, the second isσ2=0.05, and the third is σ2=0.1. Figure 5 shows the simulated cone with additive Gaussian noise.Figure 5
The simulated cone with additive Gaussian noiseσ2=0.05 (a, b) and σ2=0.1 (c, d), with 30 frames (a, c) and 60 frames (b, d).
(a)
(b)
(c)
(d)The traditional SFF, which simply uses a Gaussian filter, the SFF.PCA, and the proposed method were applied to remove these noise effects, recover the 3D shape, and compare it with respect to the above two matrices. Table1 presents the robustness of the proposed algorithm compared with that of the others. In the table, it is clear that our approach significantly improved the results in comparison with the other methods. In a severe noise situation, when σ2=0.1, the RMSE is improved by 30% and 38% compared to the traditional SFF and SFF.PCA, and the correlation is improved by 99% and 139% compared to the traditional SFF and SFF.PCA, respectively.Table 1
Results of RMSE and correlation.
Method
Variance of Gaussian noise
0
0.05
0.1
Traditional SFF
RMSE
8.9320
15.607
35.384
Correlation
0.8629
0.6131
0.0875
SFF.PCA
RMSE
8.1961
8.4943
31.220
Correlation
0.9453
0.9073
0.1097
Proposed
RMSE
7.0132
8.0385
21.912
Correlation
0.9691
0.9435
0.2625


## 3.3. Qualitative Analysis

Figure6 shows the restored 3D shapes of the simulated cone. The first row is the traditional SFF, the second is SFF.PCA, and the third one is the proposed method. The first column is the noise-free situation, the second column isσ2=0.05, and the third one is σ2=0.1. In all the cases, it is shown that the results of the proposed algorithm have a smoother surface compared to that of others. Figure 7 shows the restored 3D shapes of the real objects with noise variance of 0.05. The first column is the traditional SFF, the second is SFF.PCA, and the third one is the proposed method. In all the cases, the proposed method excels in comparison with the others. It is shown that the results of the proposed algorithm have a smoother surface compared to that of the others.Figure 6
Restored 3D shape of the simulated cone. The variances are 0 (a, d, g), 0.05 (b, e, h), and 0.1 (c, f, i). Traditional method (a–c), SFF.PCA (d–f), and proposed method (g–i).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)Figure 7
Restored 3D shape of real objects with noise variance 0.05. Real cone object (a–c), TFT-LCD cell (d–f), and Lincoln head on US (g–i). Traditional method (a, d, g), SFF.PCA (b, e, h), and the proposed method (c, f, i).
(a)
(b)
(c)
(d)
(e)
(f)
(g)
(h)
(i)


## 4. Conclusions

For application in precision agriculture, a low-cost SFF system to recover accurate 3D information is developed. The conventional SFF system is weak for noise. In this study, a novel technique to improve the shape accuracy, which employs the ICA to reduce noise and enhance the edge components, has been proposed. First, the raw image passed through vector population and normalisation. Next, with the proposed approach, the signal was decomposed to independent components based on the fixed-point FastICA, and the noise components of the signal were reduced by the shrinkage function. Then, the edge components of the signal were enhanced by removing the common component from the basis vector. Finally, the depth map was restored using the first principal component of the signal. The experiments were conducted on a simulated cone object and three real objects, using three different noise situations, thus ensuring the robustness of our algorithm. The experimental results proved that the proposed method improves the accuracy compared with the previous SFF.



---

*Source: 1859512-2020-02-19.xml*