PhD Thesis Presentation

Studies on 3D-based plant
phenotyping by multi-scaledata fusion
マルチスケールデータ融合による植物表現型の３次元計測に関する研究
王浩舟 (Haozhou WANG)
農学国際専攻（IPADS）
指導教員

加藤洋一郎 教授

Supervisor Prof. Yoichiro KATO

2023/09/23

CONTENT
01

Introduction

Close-range
02 pipeline

Cross-scale
04 data-fusion

05

Aerial
03 pipeline

Conclusion

01

Introduction

1.1 General background
Accurate and comprehensive data collection helps effective
decision-making in agriculture[1]

Traditional approach hard tomeet such demand

High-throughput
phenotyping technologies[1] https://www.forbesindia.com/fbimages/900x600/proportional/jpeg/blog/wp-content/uploads/2023/01/Agriculture-is-a-potential-solution-to-meet-food-and-climategoals.jpg

4

1.2 Limitation of existing methods

Aerial approach(distance to object > 5m)

Close-range approach(distance to object < 2m)

Survey entire farmland
efficiently

Low quality oforgan structure

Obtain better organ
structure

Low efficiency forsurveying entire farmland51.3 Research objective

Can we obtain high-quality organ
structure of entire farmland efficiently
by fusing both approaches?

6Close-range
02 3D pipeline72.1 BackgroundTraditional close-range 3D phenotyping pipeline

Image collection

3D reconstruction

Organ 3D model

Traits calculation

8

2.2 Challenge
Traditional method cannot obtain complete organ structure

Obtained

Expected
9

2.3 Solution
Implement an automatic data collection and 3D reconstruction pipeline

Automatic imagecollection

Dual rotation
reconstruction pipeline102.3 Results - obtained high-quality 3D model

Real world photo

Obtained 3D model

Data pool of 3D highquality broccoli heads
112.3 Results - 3D data processing
Calculate 3D-based morphological traits
1D2D3DVisualizationTraits

Unit

Crown/head height (m)mCenter point (x, y)mCentroid point (x, y)mRoundness-Minimum area rectangle (width, length)m

Ellipse axis length (long, short)mEllipse orientation

degree2D convex area

cm2

Projected area

cm2

3D Convex volume

cm3

3D Concave volume

cm3

Head 3D
data pool
As model attributes

Final output traits list12

2.3 Results - traits accuracy validation
Compare the shortest and longest length(hard to do manual measurements for 2D and 3D traits)13

2.4 Conclusion◼ Obtained the high-quality and complete 3D models◼ Calculated the 3D-based traits and validated accuracies◼ Built a data pool for high-quality broccoli head 3D models

14

Aerial 3D
03 pipeline

3.1 BackgroundTraditional aerial phenotyping pipeline

Digital ortho-mosaic(DOM)

Digital surface model(DSM)

Collect raw UAV
images3D reconstruction
3D point cloud(PCD)

Organsegmentation

Morphologicaltraits calculation

16

3.2 Challenges
1. Need to analysis huge amount of image data
(difficult to process in time)
Just for 1 ha200+

One image
One flight(2.7GB+)

5742 x 3648 pixels

One mapLarge amount of time-series data
(38 flights in 2022)(1GB+)

200 x 20 billion=4 trillion pixels
per flight

0.3 trillion
pixelsper flight19572 x 17664 pixels
17

3.2 Challenges
2. Hard to achieve the quality for organ-level analysis from aerial reconstruction

seamline distortion
3D canopy model(PCD)

ghost effects

2D field map(DOM)
18

3.2 Challenges
3. Complex natural environment conditions makes segmentation tasks difficult(deep learning needs large number of training data)

Huge differences between time, sunlight, soil condition, growing stage, cultivars
193.3 Solutions for analyzing huge amount of image data
Temporal data fusion
narrow the processing regions by using prior knowledge of agriculture

Broccoli head position is almost thesame as its seedling position

Narrow the processing areaaround the seedling area

(100 x 100) pixels x 3000 count = 30 billion pixels per flight ~ one raw image
per crop5742 x 3648 ~ 20 billon pixels
20

3.3 Solutions for not enough aerial quality
Spatial data fusion
combine raw images (pixel coordinates) with field maps (geo coordinates)(2341,1492)

Pixel coordinatesraw

pixel-coordinates

Better quality
Lacks spatial context

Spatialfusing(35.7393N,139.5414E, 96.34m)

Geo coordinates

geo-coordinates

Lower quality
Has spatial context
21

3.3 Solutions for lacking training data
Deep learning data fusion
low labor cost for training data annotation
pre-trained models

Broccolidataset(a) Transfer learning(b) Data augmentation(c) Active learning22

3.4 Results – temporal & spatial data fusion

Seeding detection by
pre-trained Yolo v5

Temporal (time-series) datafusion during growing stages

Head Segmentation
results23

3.4 Results - traits calculation
For each broccoli headbroccoli
center points24

3.4 Results - traits accuracy validation
Has acceptable correlation with manual measured head size

Y:Manual fieldmeasured

X: Aerial measured
25

3.5 Conclusion◼ Developed temporal data fusion method with prior knowledge of
agriculture to dramatically save the computation cost
◼ Developed spatial data fusion to improve the organ-level imagequality◼ Developed deep learning data fusion to decrease the workload oftraining data annotation for head segmentation◼ Improved 2D-based traits of broccoli head and validated bymanual measurement
26

Cross-scale
04 data fusion

4.1 BackgroundPart 2: close-range 3D pipeline

Head 3D
data poolPart 3: aerial 3D pipeline

Head shapes& positions inentire farmland

Part 4: cross-scale data fusion

High quality
head models ofentire farmland28

4.3 Solutions for cross-scale data fusion
Put the templateback to field model
Chapter 3

Lmin'

2D-basedtraitsLmax'

Broccolis from aerial

Head 3D
data pool
Chapter 2Sizetransform

Calculatedifference
123
45…

Lmin
Lmax'

3D-basedtraitsAdaptedtemplate

Pick out one withsmallest differences29

4.4 Results - cross-scale data fusion

Aerial segmentation results

Aerial field 3D models

Data fusion results30

4.4 Results - cross-scale data fusion31

4.5 Conclusion◼ Calibrated the shape errors caused by occlusion◼ Selected the calibration model automatically by Auto-ML◼ Developed data fusion workflow to place the best match fromclose-range data pool back to aerial field model

32

05

Conclusion

5.1 Highlights

Chapter 2◼ Built a close-range high-quality data pool with 3D-based

morphological traits for 189 broccoli heads
Chapter 3◼ Decreased the aerial image processing workload and improved

the organ-level accuracy by temporal and spatial data fusion
Chapter 4◼ Developed aerial & close-range data fusion to place high-quality 3D models back to field.

34

5.2 Future work

Explore the deep learning 3D
reconstruction approach, like NERF [1](faster and better quality)

Improve the data fusion approachby procedural modeling forcomplex plant structures[1] https://www.matthewtancik.com/ner

35

Achievements
JournalsWang, H., Duan, Y., Shi, Y., Kato, Y., Ninomiya, S., Guo, W., 2021. EasyIDP: A python package for intermediate data processing in UAV-based plant phenotyping. RemoteSensing 13, 2622. https://doi.org/10.3390/rs13132622 (Published)
Wang, H., Tang, L., Nishida, E., Fukano, Y., Kato, Y., Guo, W., Drone-based harvest data prediction can reduce on-farm food loss and improve farmer income Plant
Phenomics. (Under review)
Wang, H., Tang, L., Nishida, E., Fukano, Y., Kato, Y., Guo, W., Virtual broccoli farmland by drone-based phenotyping and cross-scale assimilation. (In preparation)

Conferences
Wang, H., Tang, L., Nishida, E., Fukano, Y., Kato, Y., Guo,W. July 3-5, 2023. Virtual broccoli farmland by fusing close-range and aerial phenotyping, Fifth International
Workshop on Machine Learning for Cyber-Agricultural Systems (MLCAS2023), Sarabetsu village, Hokkaido, Japan. (Oral)
Wang, H., Tang, L., Nishida, E., Fukano, Y., Kato, Y., Guo, W. Sept 27-30, 2022. Estimate Optimal Harvest Time by Cross-scale Assimilated Digital Broccoli Farmland, 7thInternational Plant Phenotyping Symposium: ”Plant Phenotyping for a Sustainable Future”, Wageningen, Netherlands. (poster)Wang, H., Tang, L., Nishida, E., Fukano, Y., Kato, Y., Guo, W. July 20-22, 2021. Cost-efficient broccoli head phenotyping using aerial imagery and SfM-based weakly
supervised learning, The 8th International Horticulture Research Conference, Nanjing, Jiangsu, China. (poster)Wang, H., Kato, Y., Guo, W., June 3-4, 2021. EasyIDP: A python package for intermediate data processing in UAV based plant phenotyping, 超分野植物科学研究会の第１回研究集会 2021, Zoom online, Tokyo, Japan. (poster)Wang, H., Kato, Y., Guo, W., May 22, 2021. EasyIDP: A python package for intermediate data processing in UAV based plant phenotyping, 農業情報学会 JSAI 2021 年次大会, Zoom online, Tokyo, Japan. (poster)36

Acknowledgement
Professors / Researchers

Dr. Yoichiro Kato (U. Tokyo)
Dr. Wei Guo(U. Tokyo)
Dr. Yuji Yamasaki (U. Tokyo)
Dr. Yuya Fukano (Chiba U.)

Dr. Wenli Zhang
Dr. Yun Shi
Dr. Yulin Duan
Dr. Feng Wang

(Beijing U. of Technology)(Chinese Academy of Agricultural)(Chinese Academy of Agricultural)(Chinese Academy of Forestry)

StaffsNaomi Morinaga, Kozue Wada, and all the technical support staffs of ISAS
Lab mates

Tang Li, Erika Nishida, Alexander Feldman, Kunihiro Kodama, Thum Chun Hau, Jaesuk IM,
Passang Wangmo, Fanyu Liu,

And all friends & family members support my life.
37Thank you✉: haozhou-wang@outlook.com