SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 3(4); March (2023), pp. 238–241
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Development of Beach Litter Detection System using Deep
Learning on Beach Clean-up
Shintaro Ogawa1, Tan Chi Jie2, Takumi Tomokawa2, Sylvain Geiser2, Sakmongkon Chumkamon2, Ayumu Tominaga3,
Eiji Hayashi2

Department of Creative Informatics, Kyushu Institute of Technology, 680-4, Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan
Department of Mechanical Information Science and Technology, Kyushu Institute of Technology, 680-4, Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan3
Department of Creative Engineering Robotics and Mechatronics Course, National Institute of Technology Kitakyushu Colllege, 5-20-1 Shii,
Kokuraminamiku, Kitakyushu-city, Fukuoka, 802-0985, Japan
1
2

ARTICLE I N FO
Article History
Received 02 December 2022
Accepted 20 July 2023
Keywords
Mask R-CNN
HTC
Field robotics
Object detection

ABSTRACTThis paper proposed a deep learning-based beach litter detector specifically designed for assessing
litter levels on beaches effectively. This litter detector was created utilizing a HTC, also known
as the Hybrid Task Cascade network, and its performance was compared to that of the traditionalmask R-CNN network in order to judge its effectiveness. The findings uncovered that the HTC
network possessed heightened sensitivity towards small and tiny litters taken within the RGB
colored images.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

Deep learning
TACO

by-pixel basis, accurately determining the type of litter
present. We validated the detector's accuracy using adataset comprising images captured during actual beachcleanup activities.

1. Introduction
Beach litter, a form of marine debris, has become asignificant issue, necessitating manual surveys that aretime-consuming. To address this, we aimed to develop alitter detection system that would minimize the labor
required for such surveys. Creating an effective litter
detector is crucial for conducting accurate and efficient
litter assessments. The range of beach litter found onbeaches varies from tiny objects like microplastics tolarger objects such as fishing goods. Moreover, there area lot of beaches that are challenging for humans to access.
In this research, we designed a litter detector utilizing
deep learning techniques for instance segmentation andobject detection for images taken on the beach. Thisdetector identifies and classifies beach litter on a pixel-2.

Debris detection system with HTC

In this research, we utilized a deep learning object
detection model namely, Hybrid Task Cascade (HTC) forthe purpose of detecting beach litter. HTC is a type ofinstance segmentation neural network model thatcombines two well-known models, the Cascade R-CNN
and the Mask R-CNN. The Cascade R-CNN is anenhancement of the Faster R-CNN, a neural network thatmeasures the extent of overlap between two regions using
Intersection over Union (IoU) with a variable threshold(represented by Equation (1)). On the other hand, Mask

Corresponding author’s E-mail: ogawa.shintaro553@mail.kyutech.jp, tan.jie-chi339@kyutech.ac.jp, m-san@mmcs.mse.kyutech.ac.jp
tominaga@kct.ac.jp, haya@mse.kyutech.ac.jp URL: http://www.kyutech.ac.jp/238

R-CNN is an object detection model that extends Faster
R-CNN by incorporating segmentation capabilities.
Therefore, HTC is formed by merging these two
established models.

Fig. 2. Sample of Dataset

Fig. 1 HTC model structure

2.1. Dataset

2.3. Discriminator creation and methods ofevaluation

The dataset utilized in this study was generated bymerging a publicly available dataset of general trashcalled TACO [1] together with images that were captured
by ourselves in Hokuto Mizukumi Park, Japan. TACO isa comprehensive dataset of various trash in the wild,encompassing a diverse range of locations beyond justbeaches, comprising a total of 1,500 images. The latter
portion of the dataset consists of 186 images specificallytaken at authentic cleanup sites. Overall, the dataset
comprises 1,686 images, with 1,636 images allocated fortraining purposes and 50 images designated forevaluation. It covers a total of 60 distinct classes. To
facilitate a comparative analysis of detection accuracy
across different datasets, we formulated and trained two
separate datasets: one employing only the TACO dataset,and the other incorporating both TACO and images
obtained from actual cleanup sites.

The discriminator was trained for a total of 2000
iterations, employing a learning rate of 0.002. The loss
function used was multi-class cross-entropy. Besides ofHTC, Mask R-CNN was also utilized during training
process to enable real-time performance comparison. The
discriminators were tested with untrained samples. To
assess the performance of the discriminator, a set of 50
untrained images was prepared, and both training models
were evaluated using these images. The accuracy wasdetermined by evaluating the F-value, recall, andgoodness of fit within the IoU range of 0.5 and above. Inobject detection, IoU serves as an accuracy measurement
metric of the expected output region, while the F-value
represents the harmonic mean of the goodness of fit andrecall. Equations (1) to (3) provided below illustrate theformulas for IoU, precision ratio, and recall ratio.

2.2. Data annotation

𝐼𝐼𝐼𝐼𝐼𝐼 =In supervised learning, the creation of accurate referencedata, known as teacher data, is essential. This process
involves mapping teacher labels to the respective target
objects within images and is referred to as annotation. For
this research, we employed the coco-annotator [2] tool toannotate each litter class, thereby generating the requiredteacher data. Figure 2 illustrates an example, with the left
image representing the original image prior to annotationand the right image displaying the annotated version. In
this particular example, a plastic drinking bottle ismanually annotated and split into distinct parts, such as
the bottle and the cap.

𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑜𝑜𝑜𝑜 𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 𝑜𝑜𝑜𝑜 𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈𝑈

𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 =𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 =𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃+𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃
𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃

𝑇𝑇𝑇𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃+𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹𝐹 𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁𝑁

(1)
(2)
(3)

3. Experimental results
Figure 3.1 and Figure 3.2 show the discrimination resultof different model trained using the datasets.

239

the addition of the dataset in the case of HTC. This raises
the question of why HTC fails to yield satisfactory results
even with the augmented dataset. One possible
explanation is the compatibility between the HTC's
model structure and the added real-world dataset. The
real
environment
datasetpossesses
distinctcharacteristics, containing a greater abundance of litter,
including numerous small objects, in comparison to theTACO dataset. As depicted in Figure 1, the model
structure of HTC optimizes the features of the Region
Proposal Network (RPN) through the pooling layer.
Consequently, the feature optimization process may have
inadvertently omitted the detection of small debris due tothe introduction of the real-world dataset. Additionally,
the classification of small objects is inherently
challenging, and misclassification may be a contributing
factor. To address these issues, several prospects can beexplored. Firstly, increasing the input resolution of themodel and elevating the resolution of input images canenhance the detection of small objects by allocating more
pixels to them. Secondly, preventing annotation ofexcessively tiny garbage and minimizing the number ofclasses during the training data creation process may help
mitigate misclassification. These approaches holdpromise for improving the performance of HTC indetecting and classifying small litter objects.

Fig. 3.1. Sample of discrimination Result(Dataset: TACO, Left: HTC, Right: Mask R-CNN)

Fig. 3.2. Sample of discrimination Result(Dataset: TACO+Real env, Left: HTC, Right: Mask R-CNN)

The experimental results were assessed based on theoverall fit rate, F-value, and repeatability for the entire
class. Additionally, the fit rate specifically for plastic
drinking bottles was examined, as indicated in Table 1.
Furthermore, Table 2 presents the label’s precisionperformance of the plastic bottle class.
Table 1. Performance of trained model

4. Summary
The objective of this research was to utilize deep learning
for the detection of debris at a beach cleanup site.
Through our investigations, we discovered that thenumber of classes in the dataset plays a crucial role.
Furthermore, we identified the need to develop effective
techniques for detecting small objects within the dataset.
Moving forward, our future endeavors will focus on
enhancing the accuracy of debris detection. We aim torefine the detection accuracy and develop acomprehensive system that can be of greater utility inreal-world cleaning scenarios. Our ultimate goal is tobuild a system that not only improves the detection
accuracy but also enhances its practicality and usefulness
in real beach cleaning sites.

Table 2. Plastic bottle label’s precision,,,Consideration
Although Table 1 indicates a relatively low precision
score, the individual class analysis for plastic bottles inTable 2 reveals better precision results. Specifically,
when considering the identification of PET bottles, Mask
R-CNN demonstrates a higher conformance rate whenreal-world environment data is added, while HTC
achieves a higher conformance rate without the inclusionof such data. Surprisingly, the overall F-measure for theclass does not experience significant improvement withReferences
1. “TACO Dataset ”, http://tacodataset.org/
2. Justine Brooks,
https://github.com/jsbroks/coco-annotator

240

Mr. Sylvain Geiser

Authors Introduction
Mr. Shintaro Ogawa
He received bachelor degree inEngineering in 2022 from intelligent
and Control Systems, Kyushu
Institute of Technology in Japan. Heis currently a Master student atKyushu Institute of Technology andconducts research at Hayashi
Laboratory.

Prof. Eiji Hayashi

Prof. Eiji Hayashi is a professor in
the Department of Intelligent andControl Systems at Kyushu Instituteof Technology. He received thePh.D. (Dr. Eng.) degree fromWaseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPerceptual information processing. He is a member of
The Institute of Electrical and Electronics Engineers(IEEE) and The Japan Society of Mechanical Engineers(JSME).

Dr. Sakmongkon

He received Doctor of Engineering
degree from Kyushu Institute ofTechnology in 2017. He was apostdoctoral Guangdong Technology
Presently he is a postdoctoral
researcher in Kyushu Institute ofTechnology since 2019. His researchinterests include factory automation robots and socialrobots.
Projected Assist. Prof. Ayumu Tominaga
Projected Assist. Prof. Ayumu
Tominaga is a professor in
Department of Creative Engineering
Robotics and Mechatronics Course
at National Institute of Technology
Kitakyushu College. He received
the Ph.D. (Dr. Eng.) degree from
Kyushu Institute of Technology in2021. His research interests include Intelligent
mechanics, Mechanical systems and Perceptual
information processing
Mr. Takumi Tomokawa
He received bachelor degree inEngineeringin2021
frommechanical system engineering,Kyushu Institute of Technology inJapan. He is currently a Masterstudent at Kyushu Institute ofTechnology and conducts researchat Hayashi Laboratory.
Mr. Tan Chi Jie

He studied Engineering andComputer Science in Ecole desMines de Nancy, France, between
September 2018 and June 2020. Heis currently a Master student atKyushu Institute of Technology,
Japan, and conducts research atHayashi Laboratory.

He received his Bachelor ofEngineering Electronics Majoring in
Robotics and Automation from theFaculty of Engineering, Multimedia
University, Malaysia in 2020. He iscurrently a Master student at Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

241