Yield Mapping in Apple Orchards
By: Dennis De Graaf
Tags: Yield Farming, Fruit Counting, Precision Agriculture, Detection, Object Detection

Yield Mapping in Apple Orchards

Introduction

In the paper[1] the researchers present a method to reconstruct an apple orchard and to estimate yield. The main focus of this Medium post is on their state-of-the-art yield mapping solution. There will be a short part dedicated to their fruit counting method. To give a quick idea of what this yield mapping is capable of, figure 1 shows an example of the reconstructed tree row.

figure 1 reconstruction of a misaligned apple tree row [1]

Yield mapping technique is necessary in precision farming and tasks like seeding, crop inspection and harvesting that need yield mapping information to work. These days there is a variety of commercial solutions available on the market. The most of these solutions are dedicated to wheat, rice and maize like crops. The most challenging crops are fruits and vegetables due to the complex shapes of orchards compared to row crops. The researchers came with a stunning solution that uses a RGB-D (a camera with 4 channels Red, Green, Blue and depth) camera.

Technical

Multi-view Fruit Tracking

Imagine a camera that records video data, is attached to a platform that drives through the orchard. When the platform is done collecting all video data, there are multiple frames with the same apple in the video data. This causes double or more countings. The researchers managed to fix this problem with a special filter technique named “Multi-view Fruit Tracking” see figure 2. This double counting filter is based on affine tracking[2] and incremental Structure From Motion (SfM)[3].

Affine tracking [2] in short, this principle is to identify points in the image which are easy to distinguish from their environment. This enables the researchers to track them. By tracking enough points, we can get an idea of the way the camera is moving through space.

Incremental Structure From Motion [3] in short, this technique determines the spatial and geometric relationship of the target through the movement of the camera, which is a common method of 3D reconstruction.

Figure 2 fruit tracking clarification [1]

Multi-view fruit merging and Yield Estimation

This technique takes the single side treeline reconstructions, and multi-view fruit counts as inputs. It merges the input reconstructions from front and back treeline sides using semantic information [4], to avoid double/over counting. This makes fruits visible from the front and back view of the tree and outputs the total fruit count for the tree row. In figure 3 the reconstructed treeline is shown.

figure 3 the final merged treeline [1]

Dataset

The researchers used six datasets, three RGB-D see figure 4 (Red, Green , Blue and Depth channels) and three RGB datasets. These six datasets contain tree row front and back sided image data. All imagery (RGB and RGB-D) data is captured using the Intel RealSense R200. The RGB-D and RGB data is used for testing the merging/reconstruction algoritm. The yield estimation used the RGB data only (see section: Multi-view fruit merging and Yield Estimation).

RGB-D

Examples of this dataset are shown in figure 4.

Dataset-I (968 images at 30 fps, 1920x1080) contains an apple-tree row of 21 trees with a lot of wild weed captured in a horizontal view. Dataset-II (2394 images at 60 fps, 640x480) contains 27 trees captured in a tilted view with a focus on tree trunks. Dataset-III (2020 images at 60 fps, 640x480) of 30 trees is collected by a camera attached to a stick in a tilted-top view of the tree canopies.

figure 4 a one side point cloud, gray color represents the depth [1]

RGB

Examples of this dataset are shown in figure 5.

Dataset-IV (873 images at 30 fps, 1920x1080) contains six trees that are mostly planar. Most of the apples on these trees (with 270 apples in total) are fully red and visible from two sides. Dataset-V (1065 images at 30 fps, 1920x1080) contains ten trees that have non-planar geometry. Apples (274 in total) in these trees are mostly red. Dataset-VI (831 images at 30 fps, 1920x1080) contains six trees that have non-planar geometry. Fruits (414 apples in total) in these trees are a mixture of red and green apples.

figure 5 frames form the RGB dataset [1]

Yield Mapping Results

As shown in figure 5, the yield mapping approach is capable of buiding well-aligned global 3D models of tree rows. The merging algoritm needs only two-sides objects (front and back view). Duplicated poles and trunks are all merged. This merging technique is validated by visual checking, if the misalignment of landmarks (e.g., poles and tree trunks) is eliminated. The accuracy of the merging are tested by comparison with manual measurements of trunk diameter and tree height. In figure 5 the yellow boxes are misalignments.