SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 4(1); June (2023), pp. 6–11
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

A Real and Synthetic Dataset for Robotic Vision in Outdoor
Beach Environment – BCRobo
Tan Chi Jie1, Takumi Tomokawa1, Shintaro Ogawa1, Ayumu Tominaga2, Sakmongkon Chumkamon1, Eiji Hayashi1

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology 680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan
Department of Creative Engineering Robotics and Mechatronics Course, National Institute of Technology Kitakyushu College, 5-20-1 Shii, Kokuraminamiku,
Kitakyushu, Fukuoka, 802-0985, Japan
1
2

ARTICLE INFO
Article History
Received 02 December 2022
Accepted 13 July 2023
Keywords
Synthetic dataset
Computer vision
Real dataset
Field robotics
Image segmentation

ABSTRACT

Datasets are one of the key elements which determine the performance of a deep learning network.
Urban environments datasets receive much attention nowadays due to the rise of autonomous cars
but off-road environment on the other hand lacks quality datasets. Offroad environments needequal attention as only 55% of the world’s population lives in urban areas. This paper tackles thisissue to close the gap of robotic visual perception on the beach, one of the common offroad
environments that lack attention by presenting a real and synthetic dataset, namely BCRobo.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

also influence the performance of a neural network.
However, regardless of how well-designed is a neuralnetwork, the models will always perform better inenvironments that are like its training dataset. In otherwords, trained models tend to skew toward their training
dataset.

1. Introduction
Deep learning is blooming since the start of the 21st
century especially in recent years with the generative
network like GPT-4 getting popular. On the other hand,
computer vision is still the center of deep learning
networks as object detection and image segmentation areproven to be extremely useful in every industry. Not tomention that electric cars are getting more and more
popular nowadays and slowly replacing the traditional
gasoline car. Largely labeled datasets like CityScapes [1]and KITTI [2] dataset contribute to the development ofautonomous driving by providing a huge learning dataset
to train computer vision-related neural networks. Apartfrom datasets, optimizer algorithms, and learning policy
neural networks trade flexibility for
accuracy in a specific territory. A simple workaround to
tackle this problem of diversity in neural networks is toThis is also known as the overfitting issue that all single
deep-learning neural networks are facing now. For
instance, an image segmentation model trained withurban datasets will have an outstanding performance inurban areas but in turn, have a corresponding substantial
performance in rural areas. Depending on the situation,
this property of a neural network might be useful because
it is specialized in doing one thing at once. In other words,
have different kinds of training datasets for different
applications.

Corresponding author’s E-mail: tan.jie-chi339@mail.kyutech.jp, m-san@mmcs.mse.kyutech.ac.jp, tominaga@kct.ac.jp,
tomokawa.takumi163@mail.kyutech.jp, ogawa.shintaro553@mail.kyutech.jp, haya@mse.kyutech.ac.jp URL: http://www.kyutech.ac.jp/

6

DK [4]. The depth camera is set to operate in Wide Frontof View (WFOV) mode, generating 1024x1024 depth
images at a rate of 15 frames per second (fps).
Simultaneously, the RGB camera is configured to capturea 1280x720 image stream in MJPEG format, maintaining
a frame rate of 30fps.

The primary objective of this paper is to drive progressin image segmentation specifically within offroad
settings, with a particular emphasis on beachenvironments. The exploration and availability ofdatasets in offroad environments, unlike urban
environments, have received less attention, primarily due
to the growing emphasis on autonomous driving cars.
However, the advancement of autonomous robots inoffroad settings, including forest and beach exploration,
remains a significant focus for researchers in the field of
robotics.

2.2. Global Positioning System (GPS)SOMA is also equipped with an Emlid Reach RS+ device,
allowing for the inclusion of GPS data in the form of
NMEA, also known as National Marine Electronics
Association messages. The GPS data is acquired with a
precision of up to +/- 5cm, utilizing Real-time Kinematic
Positioning (RTK) technology.

The paper commences by providing an overview of thesensor setup, dataset configuration, and the process ofcollecting the data. Subsequently, it presents the dataset's
statistics and evaluates its performance using three
cutting-edge image segmentation networks.

2.3. Lidar sensor
As depicted in Figure 1, a Velodyne VLP-16 rotating 3D
laser scanner is mounted on the upper part of SOMA.
This sensor has the ability to capture 3D point clouds
within a 360-degree field of view, covering a range of 1mto 100m. It operates at a rotation rate of 5Hz andmaintains an accuracy of +/- 3cm [5].

2. Sensor Setup

3. Structure of Dataset
The BCRobo dataset comprises both real and synthetic
beach environment data, including image sequences and
LiDAR 3D point clouds recorded by SOMA Robot. The
robot's operations were controlled manually by a humanoperator to travel and capture the scenes of various
beaches around Kitakyushu City and Munakata City inJapan. In total, 6850 color images were recorded, andmanually annotated ground truth images were provided
for every tenth frame of a video sequence. In cases where
the tenth frame appeared blurred, the preceding or
subsequent frame may have been used as a substitute.
The exploration data can be categorized based on thefollowing locations:Figure 1: SOMA Robot’s Sensor Setup
Figure 1 illustrates the comprehensive sensor setup
employed for capturing the BCRobo dataset. The study
utilizes SOMA, an autonomous forest and beach
exploration robot developed within Hayashi Laboratory[3]. Originally an All-Terrain Vehicle (ATV), SOMA
underwent modifications to transform into anautonomous driving exploration robot. In addition to its
innovative fully automated steering mechanism, therobot is equipped with various distinct sensor types,
listed as follows:• Jinoshima Island – An island in Munakata City witha port area and rock bed environment.
• Agawa Hosenguri Seaside Park – A typical beach forvacation and sea bathing.

2.1. RGB-D sensor
Positioned at around 1.1m above the ground, an RGB-D
sensor is mounted in the frontal area of SOMA. The
RGB-D camera employed is the Microsoft Azure Kinect
7

utilized to perform automatic annotation which generates
the annotated images based on the objects’ material type.
This dataset consists of 300 synthetic images and its
corresponding annotated images which is approximatelyaround 30% of the whole dataset.
3.1. Statistical Analysis of BCRobo Dataset

Figure 2: Class labels, Video sequences frame andannotated sample

The distribution of class annotations in the BCRobo
dataset is illustrated in Figure 5. As anticipated, thedataset exhibits a bias towards the sky, grass, sand, andwater labels. This outcome aligns with the dataset's
specific purpose of addressing the scarcity of image
segmentation datasets for beach environments. Given therequirement for autonomous robots to navigate
accurately in beach environments, it becomes essential
for them to distinguish between various types oftraversable terrain such as sand, grass, and mulch, while
identifying untraversable areas like water and rock beds.

Figure 2 depicts sample RGB images captured in Agawa
Hosenguri Seaside Park and Jinoshima, along with theircorresponding manually annotated images with 24 classlabels. Approximately 10 minutes of data were recorded
for each location, with a video frame rate of 15Hz as well
as a LiDAR point cloud rate of 1Hz. It should be notedthat not every single image in the video sequence isincluded in this dataset. Also, the annotated images
within this dataset encompass 24 classes derived from the
RUGD and KITTI datasets.

Figure 3: Data collection route predicted with GPS
Figure 5: Annotated Class Pixel Percentages

Thanks to the RTK GPS sensor, the precise location of
the robot is accurately tracked throughout the entire
recording process. However, there were instances when
the GPS connection with the base station was weak,leading to the prediction of robot routes using theavailable GPS coordinates, as illustrated in Figure 3. In
Figure 3, SOMA is controlled manually according to the
red route in a back-and-forth manner.
The synthetic part of this dataset shown in Figure 4 isgenerated through a beach environment simulated in
Unreal Engine 4. An unreal plugin, UnrealCV [13] is

4.

Experiment and Evaluation

To assess the dataset's quality and practical applicability,
three state-of-the-art semantic segmentation methods
were chosen to be trained using the BCRobo dataset.
Specifically, the selection of these three models was
based on their ResNet50 backbone structure [6] within
this experiment. ResNet50 was selected as the constantvariable in this experiment due to its significance as thepioneering successful deep feedforward neural network.
Despite being introduced as the earliest working modelof its kind, ResNet50 continues to be widely employed asa backbone and remains one of the most frequently
referenced neural networks in image segmentation
methods since its victory in the ImageNet competition of
2015. These semantic segmentation approaches selected
for this experiment are as follows:

Figure 4: Synthetic RGB and Annotated Images
8





PSPnet [7] – ResNet50 – d8 backbone
OCRnet [8] – ResNet50 – d8 backbone
UPerNet [9] – ResNet50

Table 1. Train set, Validation set, and Test set.

PSPnet, an early image segmentation approach thatincorporates global context information in scene parsing,
emerged as the winner in the PASCAL VOC andCityscapes benchmark back in 2016. The inclusion of
global scene category analysis has proven to be valuable
in intricate scene parsing scenarios, particularly in beachenvironments, where careful attention is required to
distinguish various sub-regions containing significantly
small or large objects.

Train
Validation
Test

Jinoshima
1.
31
5
39
39

Agawa

Total

%

233
30
29

548
69
68

80.00
10.07
9.93

The training processes of this experiment are conductedin the environment as below:

Nvidia RTX 3090 – 3 units
AMD Ryzen Threadripper 3960X 24-Core
MMSegmentation v0.29.1 [10]Ubuntu LTS 20.04

Among the three selected approaches for this experiment,
OCRnet stands out as the most recent one. Its notablefeature is the capability to discriminate between
contextual pixels belonging to the same object class andthose belonging to different object classes. Furthermore,
the utilization of dilated convolutions for multi-scale
context enables OCRnet to leverage the high-resolution
and extensive contextual information in this dataset.

The images are initially resized to 688x550 before being
fed into the deep learning network for training. A crop
size of 300x375 is set for the training process. The batch
size is configured at 6 per GPU, totaling 18 batches since
3 GPUs are utilized. For optimization, the Stochastic
Gradient Descent (SGD) optimizer with momentum [11]is chosen, with a learning rate of 0.015 and a momentumof 0.9. A weight decay of 0.0004 is applied. To prevent
overtraining at the beginning of the training process, the"Polynomial learning rate" policy with warmup isemployed. This policy involves linearly increasing thelearning rate for 1000 iterations before it reaches 0.015,
followed by a polynomial decay until it reaches theminimum learning rate of 0.0001 for the entire training
duration. Using MMSegmentation, the models aretrained for 2000 epochs which are approximately 60000
iterations for all three models mentioned before.

In contrast, UPerNet aims to integrate multiple tasks suchas texture recognition, object classification, pixel-level
scene parsing, and scene recognition within a single
neural network, utilizing an unprecedented learning
approach namely unified perceptual parsing. Like PSPnet,
UPerNet incorporates the Pyramid Pooling Module(PPM), applying one PPM for scene, object, part, andmaterial recognition.
4.1. Experimental Setup
Similar to other semantic datasets, the recorded images
in this experiment are divided into the train, validation,and test sets without considering the synthetic images
generated using Unreal Engine. Specifically, 80% of themanually annotated images are allocated to the train set,
while the remaining 20% is evenly distributed between
the validation and test sets. Since the dataset
encompasses two distinct beach sceneries, it is essential
to ensure that the image segmentation models are trained
equally with a similar splitting ratio. To achieve this, theaforementioned splitting ratio is applied separately toeach beach environment, and the resulting subsets are
combined as the final train set, validation set, and test set,
as shown in Table 1.

4.2. Result Evaluation
The efficiency and performance of the models areassessed using the standard semantic segmentation
metrics, including mean pixel-wise classification
accuracy (mAcc) and mean Intersection-over-Union(mIoU). mIoU is computed as the mean Intersectionover-Union of each class, where IoU is defined as
TP/(TP+FP+FN) [12]. Here, TP represents true positives,
FP stands for false positives, and FN denotes false
negatives. Additionally, mAcc is calculated as the mean
pixel classification accuracy (aAcc) across all classes.
This evaluation process involves passing the test andvalidation sets into the trained models, also known as theinferring process. The results are presented in Table 2.
Furthermore, an additional evaluation is conducted on all
three sets combined, as indicated in Table 3.
9expense of diversity. Therefore, it is advisable tocombine this dataset with others when conducting image
segmentation in scenes that extend beyond beachenvironments.

 Table 2. Testing on Test set + Validation set

mIoU
mAcc
aAcc

PSPnet, %
2.
73
.90
81.74
98.09

OCRnet, %UPerNet, %

74.64
83.26
98.06

75.34
84.22
97.83

Currently, the BCRobo dataset primarily focuses onbeaches located in the southern region of Japan,
specifically the Kyushu area. However, there are plans tofurther enhance the dataset by incorporating additional
images and manually annotated images from various
beach environments across other parts of Japan and Asia.
The dataset can be accessed and downloaded fromhttps://github.com/chijie1998/BCRobo-dataset,providing 685 real manually annotated images as well as
300 synthetic annotated images along with its original
RGB images. Due to their large sizes, complete video
sequences and point clouds are available upon request.

 Table 3. Testing on the whole dataset
including the training set

mIoU
mAcc
aAcc

PSPnet, %
3.
71
.76
78.22
98.20

OCRnet, %UPerNet, %

72.32
79.71
98.12

71.70
79.14
97.86

In general, the evaluation results indicate thatregardless of the sets used, all models achieve acommendable mIoU rate above ~70%, demonstrating
their ability to correctly learn the visual classes. The
pixel-wise classification accuracy (aAcc) also exhibitshigh values, approximately ~98%. However, somedegradation is observed in terms of mAcc. Thisdegradation is likely attributed to the presence of
irregular boundaries, which are common in beachenvironments due to factors such as fluctuating watertides, shifting sand formations, and the movement of tree
branches and leaves caused by windy conditions.

6. References
1.

2.
3.

5. Summary

4.

In summary, the BCRobo dataset is a specialized
collection of high-resolution images of beachenvironments recorded by the SOMA field exploration
robot. Consequently, every image segmentation modelthat is trained with the BCRobo dataset would likely
achieve better mIoU values if compared to otherprominent datasets. This bias towards major class labelsin the beach environments, including sky, water, sand,and trees, within the dataset contributes to this trend.

5.
6.
7.

In conclusion, the experimental evaluation using PSPnet,
OCRnet, and UPerNet demonstrates the effectiveness of
this dataset for performing image segmentation in beachenvironments, with mIoU scores exceeding ~70%.
However, it should be noted that the high mIoU achievedby these models implies that they may not perform as
well in environments other than the beach. This
limitation is a common trade-off in current neuralnetwork models, where accuracy often comes at the

8.

9.

10

M. O. S. R. T. R. M. E. R. B. U. F. S. R. a. B. S. M. Cordts, "Thecityscapes dataset for semantic urban scene
understanding," in IEEE Computer Vision and
Pattern Recognition, 2016.
P.L.C. S.a.R.U. A.Geiger, "Vision meets robotics: The
kitti dataset," in International Journal of Robotics
Research, 2013.
E.H.R.F. N. Takegami, "Environment map generationin the forest using field robot," in Proceedings ofInternational Symposium on Applied Science, 2019.
S.M.B.T.T.E.S.W.O.A. A.P.J.G.M.F.V.R.e.a.C.S. Bamji,"Impixel 65nm bsi 320mhz demodulated of image
sensor with 3µm global shutter pixels and analog
binning", in IEEE International Solid-State Circuits
Conference - (ISSCC), 2018.
J. R. Kidd, "Performance Evaluation of the
Velodyne VLP-16 System for Surface Feature
Surveying," University of New Hampshire, 2017.
X.Z.S.R.a.J.S.K.He, "Deep residual learning for image
recognition.," in IEEE Conference on Computer
Vision and Pattern Recognition, 2016.
J. S. X. Q. X. W. J. J. Hengshuang Zhao, "Pyramid Scene
Parsing Network," in Computer Vision and Pattern
Recognition, 2017.
A. G. N. A. a. J. G. V. Gupta, "OCRNet - Light-weighted
and Efficient Neural Network for Optical Character
Recognition," in IEEE Bombay Section Signature
Conference (IBSSC) 2021, 2021.
T. a. L. Y. a. Z. B. a. J. Y. a. S. J. Xiao, "Unified perceptual
parsing for scene understanding," in Proceedings ofthe European Conference on Computer Vision(ECCV), 2018.

Projected Assist. Prof. Ayumu Tominaga
He is a professor in the Departmentof Creative Engineering Robotics
and Mechatronics Course at the
National Institute of Technology
Kitakyushu College. He received the
Ph.D. (Dr. Eng.) degree from
Kyushu Institute of Technology in
2021. His research interests include
Intelligent mechanics, Mechanicalsystems, and Perceptual information processing.

10. M.
Contributors,"OpenMMLab
Semantic
Segmentation Toolbox and Benchmark," 10 7 2020.
[Online].
Available:
https://github.com/openmmlab/mmsegmentation
11. Y. G. Y. Yanli Liu, "An Improved Analysis of
Stochastic Gradient Descent," in 34th Conference
on Neural Information Processing Systems(NeurIPS), Vancouver, Canada, 202.
12. A. Rosebrock, "Intersection over Union (IoU) for
object detection," pyiamgeserach, 7 Novemeber
2016.
[Online].
Available:
https://pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/
13. Weichao Qiu, "UnrealCV: Virtual Worlds for
Computer Vision," pyiamgeserach, 27 October
2017.
[Online].
Available:
https://dl.acm.org/doi/pdf/10.1145/3123266.31293
96

Dr. Sakmongkon Chumkamon
He received a Doctor ofEngineering degree from the
Kyushu Institute of Technology in2017. He was a postdoctoral
researcher at Guangdong Universityof Technology in 2017-2019.
Presently he is a postdoctoral
researcher at Kyushu Institute ofTechnology since 2019. His research interests include
factory automation robots and social robots.

Authors Introduction
Mr. Tan Chi Jie

He received his Bachelor ofEngineering Electronics Majoring in
Robotics and Automation from theFaculty of Engineering, Multimedia
University, Malaysia in 2020. He iscurrently a Master’s student at Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

Prof. Eiji Hayashi

He is a professor in the Department ofIntelligent and Control Systems at
Kyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems, andPerceptual information processing.
He is a member of The Institute ofElectrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).

Mr. Takumi Tomokawa
He received a bachelor’s degree inEngineering in 2021 from mechanicalsystem engineering, at Kyushu
Institute of Technology in Japan. He iscurrently a Master’s student at
Kyushu Institute of Technology andconducts research at Hayashi
Laboratory.
Mr. Shintaro Ogawa
He received bachelor’s degree inEngineering in 2022 from Intelligent
and Control Systems, Kyushu Instituteof Technology in Japan. He is currently
a Master’s student at Kyushu Instituteof Technology and conducts research at
Hayashi Laboratory.

11