Utokyo Field Phenomics Lab

Virtual broccoli farmland byfusing close-range and aerial
phenotyping

Haozhou Wang
Tang Li, Erika Nishida, Yoichiro Kato, Yuya Fukano andWei Guo
2023/07/17✉: haozhou-wang@outlook.com

01

Introduction

1. Introduction
Farmland monitoring
Conventional method

For agriculture activities, it isnecessary to monitor the cropstatus• Response to stress, likedisease, pest, etc. in time.
• Decide the optimal harvest date
Limits: Time costly, labor intensive,
low accuracy & efficiency

1. Introduction
Aerial survey

The resolution is not enough for accurateorgan-level analysis

2D field maps

Drone-based phenotyping approach

Helps to collect image data for
entire field in a few hours(high efficiency)

3D crop models

Leaf occlusion(Low quality)
Poor organ structure4

1. Introduction
Close-range survey

Close-range (indoor) reconstruction canobtain ultra high-quality crop models• Need destructive sampling• ~10min processing per plant• Not suitable for surveying allplants in entire field(Low efficiency)

51. Introduction(high efficiency & quality)

Research question(high efficiency)(Low quality)
Aerial survey

3D high quality crop models ofentire field (virtual farmland)(high quality)(low efficiency)
Close-range survey

Predict yield, harvest date andincome more accurately
61. Introduction
The proposed data-fusion workflow
Template matching and transformation

ivetcru ngts
De mplisa

New method
to get HQ 3D

Close-range
AerialBroccoli field

Automatic
process

Time-series
field model

3D traits
calculateAuto-MLmodelsManualvalidate

Improved 2D
traits calculate

Calibratedtraits

7

Close-range
02 3D pipeline

82. Close-range 3D pipeline
Broccoli head 3D reconstruction

1. Semi-automatic image collection
9

2. Close-range 3D pipeline
Broccoli head 3D reconstruction

2. Image preprocessing by labor-saving dual
deep learning approaches(remove background effects)(3D reconstruction)

10

2. Close-range 3D pipeline
Broccoli head 3D reconstruction

Real world photo

Obtained 3D model11

2. Close-range 3D pipeline
Top direction correction“Lying” Coordinate

122. Close-range 3D pipeline
Top direction correction“Lying” Coordinate

Corrected Coordinate132. Close-range 3D pipeline
Traits calculation(c) Traits calculation
1D2D3D(9) Crown traits visualization

Traits

Unit

Crown/head height (m)mCenter point (x, y)mCentroid point (x, y)mRoundness-Minimum area rectangle (width, length)m

Ellipse axis length (long, short)mEllipse orientation

degree2D convex area

cm2

Projected area

cm2

3D Convex volume

cm3

3D Concave volume

cm3(10) Obtained Morphological traits
142. Close-range 3D pipeline
Traits validation

Achieve high-correlation withthe manual measurements

15

Aerial 3D03 pipeline

3. Aerial 3D pipeline
Field data collection and analysis(3D reconstruction)

Weakly supervised
segmentation pipeline(labor saving)

Head segmentation results

17

3. Aerial 3D pipeline
Morphological traits calculation
For each broccoli head18

3. Aerial 3D pipeline
Traits accuracy validation

Has acceptable correlation withmanual measured head size

Error source

Poor quality
Y:
Manual fieldmeasuredIncomplete
segmentation

X: Aerial measured

Leaf occlusion
19

Cross-scale
04 data fusion

4. Cross-scale data fusion
Model calibration by auto-machine learningCgen-rae
losPaireddatax189
AerialDestructive sampling

AerialOther broccolis

High-quality
traits

Low-quality
traits

Low-quality
traitsAuto-ML

Multi-output
regression model

High-quality
traits(calibrated traits)4. Cross-scale data fusion
Performance of Auto-ML calibration
The Auto-ML calibration improved
the traits closer to actual size

Incomplete
segmentation

Leaf occlusion

Error source of underestimation
22

4. Cross-scale data fusion

How to put high-quality
close-range models backto low quality field model?

23

4. Cross-scale data fusion
Template matching between aerial and close-range models
Put the template back to field model
head position

Lmin'calibrated
traitsLmax'
Sizetransform

Broccolis from aerial

Lminx189

123
45…

High quality
3D traits

Calculatedifference

Broccoli from close-range Template database

Lmax'

Adaptedtemplate

Pick out one withsmallest differences244. Cross-scale data fusion
Template matching results

Aerial segmentation results

Aerial field 3D models

Template matched models
25

4. Cross-scale data fusion
Template matching results

26

Discussion &05 Conclusion

275. Discussion & Conclusion
We obtained the 3D structure model andcalculated the morphological traits of broccoli
head from aerial and close-range
Implemented the virtual broccoli farm by fusing
the model data from aerial and close-range
For future work
Implement a more user-friendly UI and apply toactual farmland
Update the template matching and transformation toshape-based rather than current numerical-based28

Thank you✉: haozhou-wang@outlook.com