# Fruit Tree Image Registration Based on Improved FAST Algorithm

Juan Feng, Lihua Zeng, Jianping Li

► To cite this version:

Juan Feng, Lihua Zeng, Jianping Li. Fruit Tree Image Registration Based on Improved FAST Algorithm. 10th International Conference on Computer and Computing Technologies in Agriculture (CCTA), Oct 2016, Dongying, China. pp.212-217, 10.1007/978-3-030-06155-5_21. hal-02180000

HAL Id: hal-02180000

https://inria.hal.science/hal-02180000

Submitted on 12 Jul 2019

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Fruit Tree Image Registration Based on Improved FAST Algorithm

Juan Feng¹, Lihua Zeng²✉, Jianping Li²

1. College of Information Science & Technology, Hebei Agricultural University,
Baoding, China
yurenmatou1011@163.com

2. College of Mechanical and Electrical Engineering, Hebei Agricultural University,
Baoding, China
zlh52103@gmail.com, ljpnd527@126.com

**Abstract:** In order to promote the efficiency and accuracy of image registration, this paper proposes an improved registration algorithm for fruit tree images acquired by a dual-sensor vision system. In the algorithm, feature points are extracted by FAST detectors from Gaussian scale space, and main orientation is determined and SURF descriptor is created with statistical method of neighbor intensity distribution, matching pairs are determined with a relative ratio method and a iterative purifying method in succession. Experimental results show that the proposed algorithm outperforms previous algorithms comprehensively.

**Keywords:** Fruit tree, Fast corner detection, Gaussian scale space, Image registration

## 1 Introduction

In the research of agricultural digitization and visualization, fruit tree modeling is of great necessity to carry out [1], which not only contribute to exploring the growth and development law of fruit tree in order to guide intelligent management, also the features of canopy organ distribution are likely to be studied quantitatively [2], which can provide harvest robots with a new technical support related to target recognition and position.

Currently, there are three common technologies of digital modeling for fruit tree, which are based on stereo vision, laser scanning and three-dimensional digital instrument respectively. However, traditional stereo vision is often affected by illumination changing in non-structural environment, laser scanning is
time-consuming and data redundancy, digital instrument is highly exact but pricy
and lack of color information [3].

The PMD Camcube3.0 is a camera based on TOF technology, has tremendous
potential in the field of machine vision recently. It can capture multi-source
images at a high frame rate for 3D construction and even target recognition.
Moreover, its SBI technology can reduce the sensitivity to illumination, and make
the camera suitable for outdoor environment. Nevertheless, due to low resolution
of image and the noise, it is recommended to combine with other sensors [4].

Image fusion is for obtaining a more comprehensive image description. Among
all steps, image registration is so crucial. Traditional registration methods are
mainly focused on those images acquired by the same imaging principle cameras
[5], but sometimes, they aren't able to realize accurate registration of
multi-sources images from complicated environment and can't satisfy real time.

In this study, apple tree was chosen as research object, an improved registration
algorithm oriented to tree images acquired by a dual-sensor vision system was
proposed, which will promote the efficiency and robustness of later modeling and
object recognition.

2 Materials and Methods

2.1 Materials

<figure><img src="image_2.png" /><figcaption>Fig.1. Dual-sensor vision system</figcaption></figure>

Fig.1 shows a dual-sensor vision system, which includes a PMD Camcube3.0
camera and Logitech C270 camera, is fixed on a tripod. Adjusting lens make two cameras aim at the same object. Experimental location was situated in an apple orchard of Xiangtang village in Beijing Changping District. Shooting distance was 1.5m. The experiment was conducted on a sunny day in the ripe period of fruit. A laptop (Intel(R) Core(TM) i5-3230 CPU, 2.60GHz, 8.00GB, Windows10) was used to activate the cameras to acquire and process those multi-source images. Given the characteristics of those images, intensity image from TOF camera and gray image changed from color image were regarded as base images, their sizes were unified as 240×240 pixels.

## 2.2 Image Feature Extraction

Corner is one of image feature for registration. The FAST algorithm has a dominant advantage in corner detection speed and operation efficiency [6-7]. Therefore, it was used in this paper. However, those corner detected by traditional FAST algorithm are not scale-invariant [8]. Therefore, an improved FAST algorithm is proposed. Firstly, original images $I(x, y)$ are repeatedly smoothed with a Gaussian function $G(x, y, \sigma)$ and then sub-sampled so as to achieve a higher level of the pyramid $P(x, y, \sigma)$. The scale space is divided into 9 octaves.

$$G(x, y, \sigma) = \frac{1}{2\pi\sigma^2} e^{-(x^2+y^2)/\sigma^2} \qquad (1)$$

$$P(x, y, \sigma) = G(x, y, \sigma) * I(x, y) \qquad (2)$$

$\sigma$ is a scale factor and presents smoothness of image.

In every octave run the FAST algorithm to detect the corner points. The algorithm can refer to the literature [9]. Non maximal suppression is applied after detecting, which can compare two neighboring detected corners and eliminate the comparatively insignificant one.

<figure><img src="image_3.png" /><figcaption>Fig.2. Feature points detection</figcaption></figure>
(a) Intensity image captured by TOF camera

(b) Color image captured by color camera

## 2.3 Feature Points description

In order to make more distinction of feature description, a statistical method based on neighborhood intensity distribution is adopted. The process includes two parts: determine the main orientation of feature points and build the SURF descriptor [9]. The former is mainly used to ensure that the detected features are rotation-invariant, the latter is used for high probability of feature matching. In a circle (radius is 6) around the feature point, Haar wavelet is applied to convolve images in same scales wavelet response in *x* and *y* direction are calculated and weighted with Gaussian. A fan window counterclockwise rotate in a particular step and accumulate inside response to get a local vector, select the largest amplitude vector as main direction of the feature point. A square neighbor (side length is 20) is established at the center of the feature point. Rotating its axes to the main direction and dividing into 4 sub-region are carried out. The value of responses and absolute responses of every sub-region are accumulated, and then form 64 dimensional description vector [11]. By normalization, it can be more robust to the light.

<figure><img src="image_4.png" /><figcaption>Fig.3. Feature points description</figcaption></figure>

(a) Intensity image captured by TOF camera

(b) Color image captured by color camera

## 2.4 Feature Matching Method

In order to decrease the possibility of one-to-more or more-to-one or false homonymy points, a relative ratio method based on first nearest and second nearest
neighbor of the descriptor is used to match homonymy points[12]. For a reliable matching, correctly matched nearest neighbor points are bound to be closer than mismatched ones. Therefore, in description space, the second nearest neighbor matching can be seen as a valid estimation of error matching density. In this paper, the upper threshold of the ratio sets to 0.65. BBF (Best bin first) algorithm based on KD-Tree (K-dimension tree) is applied to accelerate the search for the nearest neighbor and improve the matching speed [13]. RANSAC (Random sample consensus) algorithm is used to eliminate abnormal pointes, which is an iterative purifying method [14].

## 3 Results and Discussion

In order to compare efficiency and accuracy of the algorithm proposed, SURF&FAST and improved FAST algorithm are carried out respectively. The manual registration are regarded as standard image template. Evaluation index of the matching effect includes running time, MSE (Mean squared error), NMI (Normalized mutual information) & COEF (Correlation coefficient).

<table><caption>**Table 1.** Comparison of experimental results
between manual registration and other three algorithms</caption><thead><tr><th>Feature<br>Extraction algorithm</th><th>Running<br>Time/s</th><th>MSE</th><th>NMI</th><th>COEF</th></tr></thead><tbody><tr><td>SURF</td><td>2.8518</td><td>0.0102</td><td>0.8324</td><td>1.0891</td></tr><tr><td>FAST</td><td>0.8454</td><td>0.0913</td><td>0.3639</td><td>0.5332</td></tr><tr><td>Improved FAST</td><td>1.0576</td><td>0.0092</td><td>0.8951</td><td>0.9691</td></tr></tbody></table>

Overall, the proposed algorithm synthetically outperforms previous algorithms, its running time is between others, and much closer to the traditional FAST algorithm. Meanwhile, improved FAST generates the smallest error and the best normalized mutual information. Which meets the principal requirements in modeling of fruit tree.

## Conclusion

In this study, an improved registration of fruit tree images was proposed. The main conclusions were as follows:
(1) Original images captured by a dual-sensor vision system could improve image registration with more various information, some of them were immune to illumination changing, which would contribute to robustness of registration.

(2) The proposed registration algorithm could tackle the problem on traditional Fast algorithm lacking scale-invariance. It also increase feature points description, which consisted of main direction establishment and SURF descriptor generation, in order to make detected points rotation-invariant.

(3) Different results showed a good performance in decreasing the running time and promoting the registration efficiency.

**Acknowledgements.** The authors thank the financial support from the Natural Science Foundation of Hebei Province (No.C2015204043) and the Science and Engineering Foundation of Agricultural University of Hebei Province (No.LG20140602).

## References

[1] Mai, C., Zheng, L., Li, M.: Rapid 3D reconstruction of fruit tree based on point cloud registration. Transactions of the Chinese Society of Agricultural Engineering. 31(supp.2),137-144 (2015)

[2] Fan, Y., Huang, X.: Research Progress in Reconstruction of Three-dimensional Structure of Trees. Forestry Machinery & Woodworking Equipment.41(2), 26-28 (2013)

[3] Liu, G., Si, Y., Feng, J.: 3D Reconstruction of Agriculture and Forestry Crops. Transactions of the Chinese Society for Agricultural Machinery.45(6), 38-46 (2014)

[4] Schulze, M.: An approach for the calibration of a combined RGB-sensor and 3D-camera device. Paper presented at the Proceedings of SPIE 8085,Videometrics,Range Imaging and Applications,XI,(2011)

[5] Ma, X., Gang, L., Feng, J., Wei, Z.: Multi-source Image Registration for Canopy Organ of Apple Trees in Mature Period. Transactions of the Chinese Society for Agricultural Machinery.45(4), 82-88 (2014)

[6] Rosten, L., Drummond, T.: A machine learning approach to corner detection. IEEE Transactions on Pattern Analysis and Machine Intelligence.32(1),105-119(2010)

[7] Wang, M., Dai, Y.P.: Local robust feature based on FAST corner detection. Beijing Ligong Daxue Xuebao/transaction of Beijing Institute of Technology, 33(10), 1045-1050 (2013)
[8] Guo, L., Li, J., Zhu, Y.: Fast image matching algorithm based on multi-scale FAST-9. Computer Engineering.38(12), 208-210 (2012)

[9] Viswanathan,D.: Features from Accelerated Segment Test. 2016-04-20 https://pdfs.semanticscholar.org/cd26/7a4b04d835dbecf01d47fc69ed3a38c23055.pdf

[10] Bay,H.,Ess A., Tuytelaars T.,Van Gool,L.: Bay H, et al. Speeded-Up Robust Features (SURF). Computer Vision & Image Understanding, 110(3):346-359 (2008)

[11]Nie,T.,Hao,X.,Fu,T.,Zhao,W.: Electronic image stabilization based on improved fast feature matching. Electronic Measurement Technology.38(11), 42-45 (2015)

[12] Feng,J.,Liu,G.,Wang,S.,Ma,X.,Zhou,W.:Multi-source Images Registration for Harvesting Robot to Recognize Fruits. Transactions of the Chinese Society for Agricultural Machinery. 44(3), 197-203 (2013)

[13] Chen, J., Han, X.: Image matching algorithm combining FAST-SURF and improved k-d tree nearest neighbor search. Journal of Xian University of Technology. 32(2), 213-217,252 (2016)

[14] Li,H.,Wang,K.,Liu,S.: Registration method between infrared and visible images of electrical equipment based on gray-scale redundancy and SURF. Power System Protection and Control. 39(11),111-115,123(2011)