2021/11/21

研究⽬的

インフラマネジメント技術国際展開研究助成・中間報告会• ケニア国 橋梁維持管理能⼒強化プロジェクト
JICAプロジェクト

ケニアにおける橋梁維持管理のためのUAVとAI損傷認識の実装

技術紹介

ケニア国道公社(道路橋梁管理者)
技術実証の実施（現地）• Ji Dang

協業

助言

プロジェクト日本人専門家（幹事会社：CTII）

埼玉大学
（党）

フィードバック• Saitama University
JICA Bridge Management System Project, Bridge Inspection & Database SWG 2021

12

全体スケジュール

本プロジェクト⽬指す成果

34

海岸道路および点検困難の橋梁

最近の進展

561

2021/11/21

Bridge Vision Inspection
Rou$neInspec$on

Performance and Eﬀec.veness

UAV BRIDGE INSPECTION
Earthquake Emergent
Inspec$on

78

High or Dangerous PlaceUAV Bridge Inspection
Girder SideRubber
Bearing
Pier

From: NEXCO

910Level of UAV Inspec0on

What UAV can inspect• Level 1: Control UAV, diagnose from pics
• Level 2: Deep learning damage detec<on• Level 3: +Auto Report Wri<ng• Level 4: +Auto Pilot• Level 5: +AI Plan (Time, Route, Ac<on)• Level 6: +Robot Repairing• -> Fully Unmanned Bride Management

11• Steel Bridge: Corrosion, Cracks, Fracture, missing bolts, strippedpainting• RC： Cracks, Spalling, Changing Collor1222021/11/21

Test(3) Wind Tunnel Test

Result(3) Wind resist V.S. PowerWind machine

OutputWindow

Wr (m/s)

P/WRp①2.5

1.95

1.05

②2.8

1.92

1.09

③6

2.29

0.776

④5

2.05

0.949

Rp=W/(P-W)

W :Weight
P: Payload

14

Result(3) Wind resist V.S. Power

Performance Index

Wr(m/s)

137
65
43
210

PI = Op + Sf +Vs
PI • Performance Index
Op • Operation performance
Sf • Safety Performance• Vision PerformanceVs

Wr=15.2-11.5Rp
0.7

0.8

0.9
Rp

1

1.1

15

16

Performance Evaluation

Performance Evaluation

17Op30

Sf35

Vs

35

Operation TimeFPV
Maximum Distance
Maximum Speed
Wind Resistant

1555525

Obstacle Avoidance
Photo Resolution

1010Video Resolution
Suspension Gimbel
IOS
Shutter Speed

105551832021/11/21

The Blue-cloud Bridge (Seiun Bashi)

2016 Kumamoto Earthquake

1920

UAV Performance evaluation for resent models

Performance Evaluation
項⽬(配点)
Opereration

M2z

Mm

21MA

⾶⾏可能時間(15)
FPV(5)
最⼤伝送距離(5)
最⼤⾶⾏速度(5)
Safety
最⼤⾵速抵抗(25)
障害物検知機能(10)静⽌画 画素Visual数(10)
動画 解像度(10)
ジンバル(5)ISO感度(5)
シャッタースピード(5)
Performanc Index P(100)

Mp

P4

M2z

Mm

Ma

97105655555554545553315122118191010101010666661010108105245245245245247671

83707422

Bridges as Inspection Objects

What UAV NEED?
Satellites (GPS)No.

Bridge TypeYear

LengthDamage Rank

123Simple Supported Steel Plate Composite Girder
Simple Supported Steel Plate Composite Girder
RC Girder + RC Pier

1963
1962
1956

(m)
40.0
29.1
236.7

33Unknown

45678910111213

Simple Supported Steel Girder
Simple Supported Steel Plate Composite Girder
Simple Supported Steel Plate Composite Girder
Simple Supported Post Tension Concrete Girder
Simple Supported Steel Plate Composite Girder
Simple Supported Steel H Girder
Simple Supported Steel Girder
Steel Arch Bridge
Simple Supported Steel Plate Composite Girder
Simple Supported Post Tension Concrete Girder

2001
1962
1968
1960
1958
1972
1970
1971
1956
1980

60.2
40.5
40.7
50.1
17.0
26.0
26.7
99.0
59.7
89.3

3322
122
33
323

What UAV NEED?
・Safe place for take off and landing
・Safe place where operator can stand・Satellites connecting
・Weather244

2021/11/21

Procedure

Total Inspection TimeUAV Inspection Flow

70
Time Cost（min）

Take off2~3m↓
Out side member photos:
Road Surface, Side of Railing, Girder
Beam Side view.
(Keep 2~3 m distance)↓Inside Superstructure Members
Surface: Girder Bottom, Inner Girder,Deck.
( Only if possible)

60
504030201000

↓

4060

80

100

Bridge Length（m）

Piers, Abutment, Walls

252026Detection Accuracy

Detection Accuracy

70

Compare with human manual inspection reports

Mean： 33.8%

60P（％）50

Σ$!
! =Σ$"

40302010# ：Percentage of UAV finds the damages that human reported

0$! ：Detect Damage Numbers by Reports

0.0$" ：Number of damages in D1 that also detected by UAV

2720.0

40.0

60.0

80.0

100.0

Bridge Length（m）

28

Steel Girder Bridge

Steel Girder Bridge

・・・UAV・・・Reports

・・・UAV・・・Reports

293052021/11/21

RC, PC Girder Bridge

RC, PC Girder Bridge

コンクリートうきCracks
Lost of cover concrete
⽀承の腐⾷

Steel Bars

・・・UAV・・・Reports

3132

Location

Autopilot base bridge inspection
⼤洋測量https://taiyo-s50.jp/publics/index/38/

UAV can confirm
UAV can not (difficult) to see

33

34

Waypoint Assignments

Waypoint Recording
Flight to the point manually

Assign some waypoints onoperation interface

Adjust location andcamera• Height• Camera Setting• SpeedSet Speed
Record thepoint

Map in the operation interface

Set all points one by one

Set all points

35366

2021/11/21

Waypoint Assignment

Onsite verification for UAV Autopilot Inspection
Length833.1m

Width

12.5m

Span

68.7m

Structure

13 Span Steel Box
Girder Bridge

J Bridge• Location：J Bridge in Saitama• Time：11:00 AM• Operator：3
Deck and Box Girder Bottom side (2xt)37

38

点検②Results

General Inspection Operators needed for Manual Inspection
Telescope
Inspection（pre）InspectionOperators
Inspection
Assistants
Inspection
vehicleoperator
Traffic ControlStaff

⾃律⾶⾏（Waypoint記録型）で撮影できた橋梁（2倍速）

39

Close Inspection

1⼈Inspection
Vehicles1⼈ 1)

OtherMachines
1⼈ 2)

2⼈

2⼈1)

2⼈

―

1⼈

1)

―

3)

2)

――

Note 1) Bridge inspection vehicles, etc .: Range required for inspection, traffic conditions, bridges and equipment used
Determine the appropriate organization personnel in consideration of the conditions.
Note 2) Other facilities: When using inspection roads, ladders, ships, painted scaffolding, etc.
Determine the number of trainees in consideration of the local situation and inspection method.
Note 3) Traffic control staff: Considering the traffic conditions at the inspection site, inspection methods, etc.
Determine the number of people to organize.

40

点検種別による利点と⽋点

For narrow spaces

⼿動制御利点

⽋点

⾃律⾶⾏
⾃律⾶⾏
（Waypoint指定型） （Waypoint記録型）
l 撮りたい箇所が撮 l 最も時間がかから l 映像が滑らか
れる
ないl 撮りたい箇所を撮l 映像が滑らかれる
l 操作⽅法が簡単
l 周囲確認の余裕がl 周囲確認の余裕が あるあるl 操作技術が求めら l 地図上で⾶⾏ルー l ⼆度⾶⾏するのでれるトを作成するので 時間がかかる(ルーl 映像がカクカクす １m弱の誤差が出 トが保存できるのるるで初回の⾶⾏のみ)
l 周囲確認の余裕が l 対象物の⾼度がわないからないと⾶⾏⾼度が設定できない

Mini Robots

Mm P=70
Mass=199g
https://prod.kyohritsu.com/av.html

41427

2021/11/21

IoT +UAV• 上向けカメラを装着し• ラズパイからWifiで繋ぐ• 指令送信で制御

For UAV images

DEEP LEARNING BASED DAMAGE DETECTION

4344

Structural Damage Featureexperience

Cat

Rememberfeatures of eachobject andremember.
ColorLine
Grayetc …

Sample data

Input

Dog

Learn the features
Predict the likelihood(Deep Learning)

Cat

Car

45

46

Structural Damage Detections DL

Data BaseV. Hoskere, Y. Narazaki, T.A. Hoang, B.F. Spencer Jr, 2017
Y. Narazaki, V. Hoskere, T.A. Hoang, B.F. Spencer Jr, 2017474882021/11/21

UAV Inspection

Accuracy

Accuracy
100%
90%80%70%60%
50%40%30%20%
10%0%

Mavic Pro
27%

*DJI

40%

Trained CNN

Data0

With
Refine

+DA02
+DA03
+DA01
With all kinds of Data Argumentation

Class+DA04

49

Image

50

Data Base

Corrosion DetectionMore in inspection reports

Corrosion

CracksSteelConcrete

Spalling
DamageLeakingFree Lime

Trees

Fans

OthersNondamage

More in real world

51

52

Data Base

Mixed Training
Reports

Inspection Reports Real world/ UAVImages
Damaged ImageNon-Damage
Image

53

A lot

Very few

Very few

A lot

Pics andDamage
Classes

Healthy
Structure
Background54

9

2021/11/21

FCN=Fully Convolution Net

Mixed Training

Conv + ReLU
Upsampling

MaxPool

Concatenate

Conv + BatchNormalization
Copy and concatenate
Copy and concatenate
Copy and concatenate
Copy and concatenate

UAV Only

Mixed Training

Augmented Mixed Training

Vgg1655

56

How to train FCN? Two methods

How to use it

Image

Squash

Train
FCN A

Annotation

Image

Squash

Drop

Crop

Input

Predict

ResizeResult A

FCN A(a) Squashing Segmentation(a) Squashing Segmentation

Image

U-Net

Train

Image

CropInput

Predict

StitchResult B

FCN B

Annotation(b) Cropping Segmentation

FCN B

57

(b) Cropping Segmentation

58

Annotation: Label damage on image picxels

FCN Semantic Segmentation

MATLAB 2019aImage Labeler

Data set

80% for training

59

20% for validationand testing

Annotations:
Undamaged pixels - index 0Damage pixels - index 16010

2021/11/21

UAV video marking

UAV video marking
Video withdamage
detection

4K Video

ConvertCombine…

…
StitchCropTest

224 × 224 pixels

Predict

FCN model

224 × 224 pixels

61

62

You Only Look Once(YOLO)

Some Preliminary Results
GroundTruth

Prediction

You Only Look Once: Unified, Real-Time Object Detection―Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi
University of Washington, Allen Institute for AI, Facebook AI Research

63

YOLO v5s

6564Architecture of Mask R-CNN

66

11

2021/11/21

Detection Result Comparison
Original Image

67

Ground Truth

Detection Result Comparison
Tensorflow+Keras

Original Image

Detectron2 (Pytroch)

Ground Truth

Tensorflow+Keras

Detectron2 (Pytroch)

68

Test on UAV Video

Thank you for listening

6970

12