E3S Web of Conferences 415, 03021 (2023)
DFHM8

https://doi.org/10.1051/e3sconf/202341503021

Real-time debris flow detection using deep convolutional neuralnetwork and Jetson Nano
Minh-Vuong Pham1, Chang-Ho Song1, Thanh-Nhan Nguyen1, Ji-Sung Lee1, and Yun-Tae Kim1*
1Department of Ocean Engineering, Pukyong National University, Busan 48513, Korea

Abstract. This study aims to develop a potential system for real-time detection of debris flow motion using
a deep convolutional neural network (CNN) and image processing techniques. A system consisting of a pretrained CNN model, NVIDIA Jetson Nano, and a camera was used to identify debris flow movement. The
pre-trained CNN model was trained on an image dataset derived from 12 debris flow videos obtained fromsmall flume tests, large flume tests, and several debris flow events. The application results of the proposedsystem on the flume test in the laboratory reached an F1 score of 72.6 to 100%. The real-time processing
speed of the CNN model achieved from 2 to 21 frames per second (FPS) on the Jetson Nano. Both theaccuracy and the processing speed of CNN model depend on the size of the video input and the input sizeof the model CNN. The CNN model of 320 × 320 pixels with a resolution of 800 × 480 pixels gives accuracy(F1 = 99.2%) and processing speed (FPS = 20) considered the optimal model when running the Jetson Nano
device; thus, it can be applied for early detection and warning systems.

1 Introduction
Debris flow are a moving mixture of loose muds, soils,rocks, and water in a steep channel that moves
downstream with a high destructive potential [1, 2]. An
early debris flow detection system is essential. To
monitor and identify debris flow movement, several
previous studies [3-9] employed sensor devices installednear torrents to measure ground vibration waves andinfrasound waves.
However, sensor alarm systems have not yet been
widely used in many parts of the world owing to the highcosts of installation, operation, and maintenance.
Recording debris flow events using a camera may
provide a potential approach to the debris flow detection
and monitoring problem. With the strong developmentof technology, the Jetson Nano device is developed byNVIDIA and is widely applied in the field of artificialintelligence. Because of its advantages such as small,
low cost, power-efficient and embedded Internet ofthings (IoT) applications. Therefore, it is very suitable
for remotely monitoring of debris flow events occurringin mountainous areas.
The objective of this study is to present a potential
method to detect debris flow motion using recorded datafrom a camera and Jetson Nano. We built a CNN model
based on the YOLO framework to detect and localize adebris flow in view of digital image recording devices.
When debris flow occurs, digital cameras installed atmonitoring stations record the images and transmit them
to a program installed on the Jetson Nano device. This
program will analyse the image and identify theappearance of debris flow on the image automatically.

The system will then give warning sounds or send
messages through the integrated system on the IoT
platform. With the help of new combined techniques ofartificial intelligence and image processing, this study
can contribute a new approach to real-time debris flow
detection for early warning and monitoring systems.

2 Methodology
The proposed approach for debris flow detection in thisstudy is illustrated in Figure 1. It consists of following
steps: (1) Prepare dataset; (2) Build and train the CNN
model to detect debris flow motion from the image
dataset; (3) Evaluate the performance of model; and (4)Apply the trained model to real-time detection.
 Data preparationxVideo collection
xAnnotation labelling

Debris flow motion detection modelx Building model CNN using YOLO frameworkx Model training

Performance evaluation

INPUT

x Accuracy evaluation
Real-time debris flow detection framework

OUTPUTx Inference speed evaluation

Fig. 1. Overview of building pre-train CNN model.

To adopt the proposed method for debris flow motion
detection and its velocity calculation, somerequirements must be met: (1) the proposed method
must have a trained CNN model to detect debris flow;(2) a digital camera must be located in the front of theflow and connected to the Jetson Nano processor; and* Corresponding author: yuntkim@pknu.ac.kr
© The Authors, published by EDP Sciences. This is an open access article distributed under the terms of the Creative Commons Attribution License 4.0(https://creativecommons.org/licenses/by/4.0/).

E3S Web of Conferences 415, 03021 (2023)
DFHM8

https://doi.org/10.1051/e3sconf/202341503021

2.2 Setup experiment(3) the proposed model only works in bright conditionand cannot work at night.

We design an experimental system as shown in Figure 3
to test the proposed system in real-time. This system
consists of 04 main parts. A small flume to produce
debris flow motion, a digital camera (normal camera orcamera pi) with a resolution of 3640 × 2160 pixels isplaced in front of the flume to record data, a program iswritten and installed on Jetson Nano to process images
and finally, a monitor connects to Jetson Nano to display
result.

2.1 Detection of debris flow motion using deep
convolutional neural network
2.1.1 Architecture of YOLO model
To identify debris flow movement from camera views.
In addition to the accuracy of debris flow detection, thecomputational speed of the proposed CNN model isconsidered a major priority. The desired model must beable to operate in real-time for monitoring and warming.
After careful examination of many CNN algorithms, weselected YOLOv4 as a benchmark model [10]. Figure 2presents the used YOLOv4 network architecture and its
detailed parameters of output features. The architectureof YOLOv4 is composed of a CSPDarknet53 backbone
to extract features [11] and a detection layer to predictdebris flow and bounding boxes.
Input image

Darknet 53

Debris flow Flume1
Flume test toprocuce debris flow

ResidualConvolutional 1×1

Convolutional setUp samplingFig. 3. Modelling real-time monitoring system of debris flowin the laboratory.

3 Dataset

Scale 226×26×1
Scale 1
13×13×1

The dataset was derived from 12 debris flow videos toprepare a dataset for training the CNN model. Five small
flume tests [15], 3 large flume tests were selected fromexperiments by the United State Geological Survey [16],and 4 recorded debris flow events occurred in theIllgraben area, Switzerland [17, 18].

Concatetenate

Predict 1×1

Predict 2

Predict 3×2×8×8×4

Residual
Convolutional 1×1
Convolutional 3×3
Convolutional 3×3

Jetson Nano

Camera Pi

Scale 352×52×1

Conv2d 1×1

Convolutional 1×1

24
Debris flow
detection

Detection layer

Convolutional layer

416× 416×3

3Processor

Convolutional set
Convolutional 1×1

Convolutional 3×3

Convolutional 1×1

Fig. 2. Structure of a You Only Look Once (YOLO) network.
(b)

2.1.2 Metrics to evaluate the performance of realtime system
Precision (Eq 1), recall (Eq 2), and F1 score (Eq 3) wereselected in this study to evaluate the performance of themodel [12, 13, 14].

Precision=

Recall=

TP
TP+FP

TP
TP+FN

F1score=2×Precision×Recall
Precision+Recall(c)(1)(d)

Fig. 4. Debris flow events occurred in the Illgraben area,Switzerland [17, 18]: (a) Illgraben area, (b-d) Debris flow
front along the channel.

(2)
(3)

4 Result and discussion
After completing the training and validation of the CNN
model, the weights of the CNN model were saved. To
evaluate the detection capability and processing speedof the five proposed CNN models (with input size ofmodel: Model 1: 288 × 288; Model 2: 320 × 320; Model
3: 416 × 416; Model 4: 512 × 512; Model 5: 608 × 608). The proposed CNN model was applied to three debris
flow videos taken from small flume tests withresolutions of 3840 × 2160 pixels, 1920 × 1080 pixels,
and 800 × 480 pixels. The model evaluation results of

where true positive (TP) indicates the number ofcorrectly detected debris flows. False positive (FP)indicates the number of incorrect detections. False
negative (FN) indicates the number of missed detections
The true negative (TN) case is not considered in theperformance assessment because there is no ground true
object in a particular region of the image.
The number of frames per second (FPS) is used toevaluate the speed at which the system can process.

2

E3S Web of Conferences 415, 03021 (2023)
DFHM8

https://doi.org/10.1051/e3sconf/202341503021

accuracy and speed on three videos were shown in Table
1 and Figure 5. Figure 6 presents examples of debris
flow motion detection on the flume test. In test 1 (video1), the proposed CNN model achieved a precision of77.6 to 100%, recall ranged from 85.2 to 100%, F1
ranged from 81.2 to 100% and processing speed ranged
from 2 to 2.8 FPS. In test 2 (video 2), the CNN model
achieved a precision of over 80.6%, recall of over88.5%, F1 ranged from 87.3 to 100% and processing
speed ranged from 5 to 7.2 FPS. In test 3 (video 3), theCNN model achieved a precision of over 62.4%, recallof over 86.9%, F1 score of 72.6 to 100%, and processing
speed ranged from 10 to 21 FPS.
The test results of three videos show that the real-timeprocessing speed of the model depends on the size of thevideo input and input size of model CNN: The higher
resolution, the slower to process. The higher size of theCNN model, the slower to process.
According to the test results on 3 different camera
resolutions and 5 different models of CNN. We see thatthe CNN model of 320 × 320 pixels with a resolution of800 × 480 pixels gives high accuracy (F1 = 99.2%) anda fast enough processing speed (FPS = 20). Therefore,
this CNN model is considered the optimal model whenusing Jetson Nano device.

(b)

(c)
Fig. 5. The result of model accuracy and speed on testing
videos. (a) Video 1: 3840 × 2160 pixels; (b) video 2:1920 ×1080 (c) video 3: 800 × 480.

Table 1. The result of model accuracy and speed on 3 videos.
Input
video

3640
×2160

1920×1080

800 ×480

CNN
ModelPrecision

Recall

F1

FPS

1

100

95.1

97.5

2.8

2

98.4

98.4

98.4

2.7

3

100

100

100

2.5

4

100

88.3

93.8

2.1

5

77.6

85.2

81.2

2

1

100

93.4

96.6

7.2

2

98.3

96.7

97.5

6.9

3

100

100

100

6.6

4

100

88.5

93.9

65

80.6

95.1

87.3

5

1

100

95.1

97.5

21

2

100

98.4

99.2

20

3

100

98.4

99.2

184

100

86.9

93

15

5

62.4

86.9

72.6

10Frame 1

Frame 2

Frame 3

Frame 4

Frame 5

Frame 6

Frame 7

Frame 8Fig. 6. Examples of debris flow motion detection on smallflume test.

(a)3

E3S Web of Conferences 415, 03021 (2023)
DFHM8

https://doi.org/10.1051/e3sconf/202341503021



5 Conclusions

on Computer Vision and Pattern Recognition(2017)
14. J. Redmon, A. Farhadi, ArXiv preprint (2018)15. M.V. Pham, Y.T. Kim, Landslides 19, 2473–2488(2022)
16. M. Logan, R.M. Iverson, M.K. Obryk, Video
documentation of experiments at the USGS debrisflow flume 1992–2017 (2018)17. M. Hürlimann, D. Rickenmann, C. Graf, CGJ 40,161–175 (2003)
18. C. Berger, B.W. McArdell, F. Schlunegger,Geomorphology 125, 421–432 (2011).

The goal of this study was to introduce a novel methodfor detecting debris flow motion. The main conclusions
drawn are as follows:
The proposed CNN model successfully detected
debris flow motion from camera view. The results
indicated that detection model using CNN achieved an
F1 score of 72.6 to 100% on the small flume test. Theaccuracy of the model depends on the size of the CNNmodel.
The real-time processing speed of the modelachieved between 2 and 21 FPS on the Jetson Nano. The
processing speed depends on the size of the video input
and input size of model CNN: The higher resolution, theslower to process. The higher size of the CNN model,
the slower to process.
The CNN model of 320 × 320 pixels with aresolution of 800 × 480 pixels gives accuracy (F1 =99.2%) and processing speed (FPS = 20) considered theoptimal model when running the Jetson Nano device.

References
1.
2.

R.M. Iverson, Rev. Geophys (1997)
O. Hungr, S. Leroueil, L. Picarelli, Landslides 11,167–194 (2014)
3. L. Marchi, M. Arattano, A.M. Deganutti,Geomorphology 46, 1–17 (2002)
4. C.J. Huang, H.Y. Yin, C.Y. Chen, C.H. Yeh, C.L.
Wang, JGR 112, 1–20 (2007)
5. I. Vilajosana, E. Suriñach, A. Abellán, G.
Khazaradze, D. Garcia, J. Llosa, NHESS 8, 805–812 (2008)
6. A. Kogelnig, J. Hübl, E. Suriñach, I. Vilajosana, S.
Zhang, N. Yun, B.W. Mcardell, A study ofinfrasonic signals of debris flows, in Processingsof the International Conference on Debris-Flow
Hazards Mitigation: Mechanics, Prediction, andAssessment 563–572 (2011)
7. H.T. Chou, Y.L. Chang, S.C. Zhang, JCIE, 36,335–347 (2013)
8. V. Coviello, M. Arattano, L. Turconi, Natural
Hazards 78, 2055–2080 (2015)
9. E. Marchetti, F. Walter, G. Barfucci, R. Genco, M.
Wenner, M. Ripepe, B. McArdell, C. Price, J.
Geophys. Res. Earth Surf 124, 567–587 (2019)
10. A. Bochkovskiy, C.Y. Wang, H.Y.M. Liao, ArXivpreprint (2020)
11. C.Y. Wang, H.Y.M. Liao, I.H. Yeh, Y.H. Wu,
P.Y. Chen, J.W. Hsieh, ArXiv preprint (2019)
12. K. He, G. Gkioxari, P. Dollar, R. Girshick, Mask
R-CNN, in Proceedings of the IEEE International
Conference on Computer Vision 2980-2988(2017)
13. J. Redmon, A. Farhadi, YOLO9000: Better, faster,stronger, in Proceedings - 30th IEEE Conference4