Underwater Platform for Intelligent Robotics and its Application

Paper: Rb30-2-8926:

Underwater Platform for Intelligent Robotics and its Applicationin Two Visual Tracking Systems
Yuya Nishida∗ , Takashi Sonoda∗ , Shinsuke Yasukawa∗∗ , Kazunori Nagano∗∗ ,Mamoru Minami∗∗∗ , Kazuo Ishii∗ , and Tamaki Ura∗
∗ Kyushu Institute of Technology

2-4 Hibikino, Kitakyushu-shi, Fukuoka 808-0196, Japan
E-mail: {ynishida@lsse, t-sonoda@brain, ishii@brain}.kyutech.ac.jp, ura@iis.u-tokyo.ac.jp∗∗ Institute of Industrial Science, The University of Tokyo
4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan
E-mail: yasukawa@liquidinc.asia, k-nagano@iis.u-tokyo.ac.jp∗∗∗ Graduate School of Natural Science and Technology, Okayama University
3-1 Tsushimanaka, Kita-ku, Okayama-shi, Okayama 700-8530, Japan
E-mail: minami-m@cc.okayama-u.ac.jp[Received October 3, 2017; accepted March 7, 2018]

A hovering-type autonomous underwater vehicle(AUV) capable of cruising at low altitudes and observing the seafloor using only mounted sensors and payloads was developed for sea-creature survey. The AUV
has a local area network (LAN) interface for an additional payload that can acquire navigation data fromthe AUV and transmit the target value to the AUV.
In the handling process of the state flow of an AUV,
additional payloads can control the AUV position using the transmitted target value without checking the
AUV condition. In the handling process of the stateflow of an AUV, additional payloads can control the
AUV position using the transmitted target value without checking the AUV condition. In this research, water tank tests and sea trials were performed using an
AUV equipped with a visual tracking system developed in other laboratories. The experimental results
proved that additional payload can control the AUV
position with a standard deviation of 0.1 m.

Keywords:
autonomous underwater vehicle, visualtracking, sea-creature sampling

1. Introduction
The seabed contains extremely rich resources such ashydrothermal ore deposits, methane hydrate, and cobaltrich crust. However, unlike the methods to extract mines
and oil fields, no profitable method to extract these resources has been established. One of the reasons for thisis that the seabed topographical structure and surrounding
environment are mostly unclarified. To conduct a seabedsurvey at a depth of 50 m or more, to which ordinary
divers cannot dive, remotely operated vehicles (ROV) and
autonomous underwater vehicles (AUV) are used. An

Journal of Robotics and Mechatronics Vol.30 No.2, 2018

Surveyurvey flexibility
High

ROV
(Remotely Operated Vehicle)

Intelligent
Underwater Vehicle

・Structure installation
・Mineral collecting ・Drilling
・Gravity core・Creature collecting
・Water component measuring
・Water temperature measuring
・ph measuring

AUV(Autonomous
Autonomous Underwater Vehicle)
Vehicle・Habitat mapping

・Topography measuring

Low
Low

High Survey area

Fig. 1. Tasks performed by underwater vehicles.

ROV receives electric power through umbilical cables andis remotely controlled by a ship whereas all the underwater operations of AUV are programmed prior to its diving
into the ocean. Fig. 1 shows the surveys and tasks conducted with ROV and AUV. The ROV, with a manipulator,
is used for surveys and performing operations such as collection of mineral and living sea-creatures [1, 2] as wellas core samples [3], drilling into a hydrothermal vent [4],and temperature measurement of water from the vent [5],as it can be operated in accordance with the underwater
environment in a flexible manner. An AUV with no constraints due to cables is used for wide area surveys suchas measurement of seabed topography and image mapping of biological community [6–8]; however, it cannot
perform tasks that require interaction with the surrounding environment. Therefore, ROV and AUV are both necessary, and hence a large operational cost is required toperform a detailed seabed survey that includes wide area
mapping of an observation target area and collection of
the target during a single navigation. To realize an efficient seabed survey, the authors have been studying an
1Nishida, Y. et al.

Fig. 2. Hovering type AUV Tuna-Sand2.

AUV that could not only perform a wide area mapping of
the seabed, but also undertake sampling or other interactive tasks with the environment. For an AUV to collectan object from a deep ocean seabed in unknown environment and securely return to the sea surface, it needs tohave a superior performance and robust software configuration. In this study, as the first step toward the realization of AUV’s automatic sampling, we developed an
AUV that can control vessel motion according to the designated speed and position while ensuring its safety on theseabed. In this work, the system configuration and kinematics of the developed hovering-type AUV Tuna-Sand2
are introduced, and a sequence of safe surveys and software configuration are proposed. We also report the results of a sampling experiment conducted using a payload
developed by Ishii and coworkers of Kyushu Institute ofTechnology; this experiment verified the effectiveness of
the AUV, and a docking experiment was also conductedusing a payload developed by Minami and coworkers of
Okayama University.

while the ground speed and altitude data from the Doppler
velocity logs (DVL), depth data from pressure sensor, anddata of true azimuth and posture from inertial navigation
system (INS) are used for underwater self-localization.
The CPU for navigation controls the thrusters and tracks aroute based on the estimated position and waypoints introduced in advance. An acoustic modem for command linkwith a function of bidirectional communication is used toreceive commands from support ships and transmit AUV
navigation information.
A camera, which captures forward images from the
AUV, is connected to the CPU for obstacle detection. A
sheet laser emits light in forward direction and the camera
captures images of the reflected light to detect any obstacle. If an obstacle lies in front of the AUV, the distanceto the obstacle is calculated by the light-section method,
and the results are transmitted to the CPU for navigation
through Ethernet. The observation equipment mounted
on the Tuna-Sand2 is a 3D mapping system [12, 13] developed by the Thornton Laboratory of The University ofTokyo. This 3D mapping system consists of a pressureresistant container, which contains a CPU, control circuit,and two LED strobes. The CPU for mapping controls thelighting interval of the LED strobes and the sheet laser; italso controls the camera to regularly capture laser reflection images and seabed images. The shape of the seabedcan be estimated from the acquired laser reflection images, and the color of the seabed can be ascertained fromthe seabed images. With these data, 3D seabed mosaic
images can be created offline. The interface is gigabit
Ethernet, through which navigation data can be received
and target values are transmitted to control the positionof the AUV. The software configurations for payloads are
explained in Section 2.4.

2.2. Kinematics

In this study, we developed AUV Tuna-Sand2 [9]shown in Fig. 2 on the basis of the technical knowledge of the hovering-type AUV Tuna-Sand [10, 11], obtained from The University of Tokyo. Tuna-Sand2 has
two pressure-resistant containers, namely, a control hull
and a mapping hull as shown in Fig. 3. The payloads
for necessary sensors and observation equipment are connected through underwater connectors.
The control hull of the AUV contains three CPUs, oneeach for navigation, obstacle detection, and 3D mapping,connected with one another through LAN. Various sensors and payloads are connected to the CPU for navigation, which controls all the information and navigation
details of the AUV. The latitude and altitude data fromthe GPS are used for self-localization on the sea surface

The hovering-type AUV Tuna-Sand2 has six 500 W
thrusters, of which four are for horizontal navigation andtwo for vertical navigation. The location and normal
thrust direction of each thruster are presented in Fig. 4.
To control motions in the surge, sway, and yaw directions and make the maximum forward thrust and the maximum backward thrust the same and the maximum starboard thrust and the maximum port thrust the same, thefour horizontal thrusters THFR , THFL , THRR , and THRL aretilted from the X axis by −θH , θH , π + θH , and π − θH ,respectively and placed on the X-Y plane. For efficientplacement of the payloads and to prevent the field of vision of the camera from being blocked by the sand dust
brought up by current, the two vertical thrusters, TVR andTV L , are placed on the Y Z-plane, and tilted from the Z-axisby −θV and θV , respectively. For static stability in the rolland pitch directions, the payloads and buoyance materials
are placed in such a way that the center of gravity andcenter of buoyancy both lie on the Z axis and the center ofgravity lies below the center of buoyancy. With the thrusts
from the thrusters shown in Fig. 4, the thrust forces, FSRG ,FSWY , and FHV acting on the AUV in the surge, sway, and2

Journal of Robotics and Mechatronics Vol.30 No.2, 2018

2. Hovering-Type AUV Tuna-Sand2
2.1. System Architecture of the AUV

Underwater Platform for Intelligent Robotics and its Application

Acoustic modem forimage transmission

Wireless LAN

Ethernet interface
for additional payload
CameraAcoustic modem
for command link

Mapping hullCPU
for mappingGPS

Sheet laser

LAN HUB
Two LED strobes
Depth sensor
CPU forobstacle detection

CPUfor navigation

DVL

Control hullINS
Profiling sonar

Six thrusters

Sheet laser

Two LED lights

CameraFig. 3. Block diagram of the AUV Tuna-Sand2.

Top view
THFR

FSRG = (−THRF − THF L + THRR + THRL ) cos θH (1)

Y

θH

θH

(lHX , lHY , 0)

THRR

(-l
-lHX , lHY , 0)

FHV = (FVR − FV L ) cos θV

X

THFL

(-l
-lHX , -lHY , 0)
θH

THRL

. (4)

MYW = (−THFR + THF L + THRR − THRL )·(lHY cos θH + lHX sin θH ) . . . . . (5)Tuna-Sand2 is not controlled in the roll direction since
the restoring moment generated by the gravity and buoyancy is larger than MRLL given by Eq. (4). In this case, theTVR and TV L are determined by FHV as follows.

Front view

(0 , lVY , -lVZ). . . . . . . . (3)MRLL = (TVR − TV L )(lVY cos θV + lV X sin θV )ΣR
(lHX , -lHY , 0)
θH

FSWY = (FVR − FV L ) sin θV+(−THFR + THFL − THRR + THRL ) sin θH (2)

θV

θV

TVR

ΣR

TVR = TV L =(0 , -lVY , -lVZ)
TVL

Y

Z

Fig. 4. Location and direction of six thrusters.

FHV
2 cos θV

. . . . . . . . . . (6)The thrusts of the horizontal thrusters are determinedfrom Eqs. (1), (2), and (5). Here we consider the thrustof the horizontal thrusters necessary to generate FSRG .
Since FSWY = MYW = 0, for the thrust of the horizontal thrusters to contribute only to FSRG , constraint conditions THF R = THF L and THRR = THRL have to be satisfied. Because the thrust ratio of the front and rear thrusters(THFR + THF L )/(THRR + THRL ) is not determined uniquelyfrom Eqs. (1), (2), and (5), we assume the ratio to be 1.
From the above constraint conditions, the assumption, andEq. (1), the following equation is derived.
−THF R = −THFL = THRR = THRL =

FSRG
. (7)
4 cos θH

heave directions, respectively, and the moments MRLL and
MYW in the roll and yaw directions, respectively, are expressed by the following equations. θH and θV need to be
between 0◦ and π /2.

Similarly, the thrust from the horizontal thrusters required to generate FSWY and MYW is calculated and added

Journal of Robotics and Mechatronics Vol.30 No.2, 20183

Nishida, Y. et al.

AUV software
Mode 80(End)

GPS taskd2

Mode 0(Wait)

d1

Abnormality
detection

Obstacle
detection

DVL task

Mode 70(Surfacing)

Arrangingfor additional payload(Mode 40 to Mode 60)

Shared memory

Mode 10(Dive)

Depth
sensor task

Mode 20(Alignment)

Mode 30(Cruising)Handling
manager

INS task

StatusserverHandling
serverThruster
controllerhAdditional payload software

Fig. 6. Software architecture for AUV handling.
Fig. 5. State flow of the AUV Tuna-Sand2.

to Eq. (7) to obtain the following:
THF R = −aFSRG − bFSWY − cMYW

. . . . . (8)THFL = −aFSRG + bFSWY + cMYW

. . . . . (9)THRR = aFSRG − bFSWY + cMYW

. . . . .

(10)

THRL = aFSRG + bFSWY − cMYW

. . . . .

(11)

Here, we havea = (4 cos θH )−1 . . . . . . . . . . . . (12)b = (4 sin θH )−1. . . . . . . . . . . . (13)c = {4(lHY cos θH + lHX sin θH )}−1 . . . . . (14)changes the mode to next mode. Mode 40 to Mode 60
are reserved for payloads. Using appropriate payloads
and mode configuration, the AUV can collect living seacreatures or perform docking of a seabed station. If there
is no payload mode or the payload process finishes, theAUV shifts to Mode 70 and begins surfacing. When it
surfaces to depth d2 or lesser, the AUV judges that it hasreached the sea surface; it switches off the DVL and payloads and finishes the session program (Mode 80). The
flow from Mode 0 to Mode 80 of the AUV is managed bya mission manager, which executes the processes sequentially.

2.3. State Flow
The operation of Tuna-Sand2 is controlled by a state
flow, which is divided into modes and each CPU executes
a process according to the mode. The state flow of TunaSand2 is shown in Fig. 5. After executing a mission program, the AUV changes its mode to Mode 0 to be readyto dive into the sea. Underwater, the AUV dives with
its weight and changes its mode to Mode 10 on reaching depth d1 . In Mode 10, DVL and other sensors as wellas the sheet laser whose operation is restricted on ground,are switched on and the altitude is monitored while the
AUV keeps diving. When the altitude becomes lower thanh, the AUV shifts to Mode 20 and begins controlling thealtitude and fixing its position. After hovering at the position for a certain period of time, the AUV changes themode to Mode 30. In Mode 30, the AUV begins navigation with a constant speed and constant altitude following the predefined waypoints and the CPU for obstruction
detection and CPU for mapping execute respective processes. When it passes by all of the waypoints, the AUV

2.4. Handling System
The CPU for navigation in Tuna-Sand2 manages modeswitching and target position in Mode 0 until Mode 30 andin Mode 70 to Mode 80 in the state flow explained in Section 2.3, and executes the processes. On the other hand,
Mode 40 to Mode 60 are assigned to the processes for
payloads, where the AUV is operated for a target object
detection by the external sensors, for example, for collecting seabed sea-creatures and performing docking of aseabed station. Fig. 6 shows the software configuration ofa handling system of Tuna-Sand2. The CPUs for payloads
or purposes other than navigation, receive the sensor datastored in a shared memory, navigation information suchas PID control, and distance to an obstacle obtained froman obstacle detection task through the status server. Target
values and other information in Mode 40 to Mode 60 are
stored to the shared memory through a handling server.
An abnormality detection task checks for any water leakage in the pressure-resistant vessels and abnormal battery
voltage and forces the AUV to surface immediately if any
abnormality is present. A handling manager manages themodes and executes the sequence shown in Fig. 7 for safe
navigation of the AUV according to the target values received through the handling server. In Mode 40, the stateof the AUV, including the power of necessary equipment
such as thrusters and DVL, and the state of communication with the payloads are monitored. If any abnormality is found, the handling process is immediately finished,

4

Journal of Robotics and Mechatronics Vol.30 No.2, 2018

With the above equations, the arbitrary strengths of
FSRG , FSWY , FHV , and MYW can be supplied to the AUV.
For the proposed AUV, we assume that FSRG , FSWY , FHV ,and MYW are independent variables and are calculated by
the PID control of the position or speed.

Underwater Platform for Intelligent Robotics and its Application

Start

Start

conditions?

Vertical
control mode?

Bad
0

1

2

3

Acquiringtarget depth

Acquiringtarget altitude

Acquiringtarget relative pos.

Good
No

Updatedhandling data ?

Converting toaltitude

Yes
What ismission mode?
41

42

Horizontal
movement

43

Verticalmovement

Rotating
movement

Have passed?

No

No

Within range?

Yes

Yes

Depth
controller

Altitudecontroller

End

End

Fig. 7. Flowchart of handling manager.

Fig. 9. Flowchart of vertical movement.

Start

and the mode is changed to Mode 70. If no abnormality
is found and the handling data are updated, the current
mode is reconfirmed, and actions are taken according to
the mode. The horizontal movement is initiated according to the flowchart shown in Fig. 8. The payload can instruct target horizontal position in absolute, UTM, or the
relative coordinate system by using the horizontal control
modes (0 to 3). If the horizontal position is designatedwith the altitude, latitude, and UTM coordinate system,
it is checked whether the AUV passes by the designatedposition in Mode 30 to ensure the safety. The designationof a target value in the robot coordinate system is limitedby the maximum movement distance. All target values
can be expressed as those in the UTM coordinate system
for horizontal position control, which allows navigation
without any erroneous operation.

The flowchart of the vertical movement is shown in
Fig. 9. The vertical movement can be designated by using
one of the three kinds of target values, namely the depth,
altitude, or the heave-direction position in the robot coordinate system. When the depth is designated, it is checked
whether the AUV has passed the global positions for horizontal movement as designated. When the AUV navigateswith the designation of the altitude or heave-direction position, the DVL may not be able to measure the altitudedepending on the distance from the seabed. If the altitude
cannot be measured, the altitude cannot be controlled. To
prevent this case, it is checked whether the designatedposition (the altitude converted from the current altitude
when the heave-direction position is designated) is within
the DVL’s measurable range. If not, the control based on
the target value is not executed. Because the depth andaltitude are measured with respective sensors having different resolutions and sampling intervals, the optimal gainis hardly obtained if the same control system is used tooperate the sensors. Because the depth and altitude aremeasured with respective sensors having different resolutions and sampling intervals, the optimal gain is obtainedwith difficulty if the same control system is used to operate the sensors. Therefore, the sensors are separately
operated using the respective control systems for the vertical movement. The rotation movement can be specified
with either of the two target values, specifically, the relative and absolute angles, depending on the control mode.
No matter which target value is used, the position is controlled with the absolute angle. Unlike the horizontal orvertical movement, no specific limit is set for the target
values of the rotation movement.
Using the above-mentioned handling system, we conducted AUV experiments in a water tank and in an actual sea with the payloads developed by Ishii’s group of
Kyushu Institute of Technology and by Minami’s groupof Okayama University.

Journal of Robotics and Mechatronics Vol.30 No.2, 2018

5

Horizontal
control mode?
0

1

2

Acquiring target
global position

3

Acquiring target
east and north pos.

Converting to
UTM coordinate

Have passed?

Acquiring target
relative pos.

Converting toeast and north pos.

Yes

Yes

No

Not so far?
No

Horizontal pos.
controller

End

Fig. 8. Flowchart of horizontal movement.

Pipe

φ130
P2

2.0 m

P6

P5

P1

Docking pole

N
P3

P0

P4
2.0 m

-6 degNorth-south position [m]3D marker

West-east position [m]Nishida, Y. et al.

-1.1
ー Target-1.2

Docking- AUV position

-1.3
-1.4
-1.5
2.1
2.0
1.9
1.8
1.7
11:12:00

AUV

11:13:00

11:14:00
time [hh:mm:ss]

11:15:00

11:16:00

Fig. 11. Position during visual tracking by 3D-MoS.
Fig. 10. Setup for docking to station.
3D marker

Pipe with 130 mm diameter

3. Application to Visual Tracking System
3.1. Docking to the Station in a Water Tank
With the payload of a stereo vision system, the “ThreeDimensional Move on Sensing (3D-MoS)” developed byMinami and coworkers of Okayama University, TunaSand2 was used to perform an experiment in which it
docked to a station in a water tank. The 3D-MoS, made by
Minami’s group, uses stereo vision to create a 3D modelof a 3D marker having three colored balls, and maps themodel to each camera image. The position and posture of
the 3D marker at which correlation between the mappedmodel and the actual images of the 3D marker is maximized are calculated by a genetic algorithm. The relative
distance to the target can be measured in real time [14,15]. Fig. 10 shows a schematic of the experimental setup.
For the experiment, we installed the 3D marker, which
3D-MoS tracked, and a station to a wall of dimensions
8 m × 8 m and 8 m deep pool. The station has a pipe, towhich a pole attached to Tuna-Sand2 can be inserted. The
3D marker has three colored balls and the stereo visionsystem 3D-MoS calculates the marker’s position online
using a genetic algorithm online and employs the highestadaptation position as the marker’s position. The diameterof the pipe to which the pole is inserted is ϕ 130 mm andthe distance from the 3D marker to the pipe is known. In
this experiment, the AUV begins navigation in Mode 30
and moves to the waypoints P0 to P5 at a depth of 4 m.
After reaching P5 , the AUV surfaces to the depth of 1.7 m
where the 3D marker is placed, and moves to P6 , at which
point, 3D-MoS can recognize the 3D marker. After reaching P6 , the AUV begins visual tracking of 3D-MoS (inMode 40). When the relative position error between the
AUV and 3D marker becomes smaller than a threshold,the AUV performs the docking operation and finishes navigation. In the experiment, the target values given by 3DMoS are the position data in the robot coordinate system.
Fig. 11 shows the north-south and east-west positions of
the AUV in the visual tracking by 3D-MoS. In the exper6

Docking poleFig. 12. Captured image by 3D-MoS at docking.

iment, the AUV was designated to move according to thetarget values from 3D-MoS not only for the horizontal position but also for the depth and direction. However, thosedata are omitted here, because when it reached P6 , the
AUV’s relative position errors in the heave and yaw directions converged below the threshold. Although there wasan overshoot of approximately 0.05 m in places where thetarget position changed significantly, Tuna-Sand2 could
follow the target values given by 3D-MoS and its position could be controlled within a final error of ±0.01 m or
smaller. When the relative position to the target became
smaller than the threshold, the AUV immediately moved
to the station and successfully performed the docking at
the station as shown in Fig. 12. Successful insertion of
the pole to a ϕ 130 mm pipe indicates that Tuna-Sand2
with 3D-MoS mounted indicates that the relative position
to the 3D marker was controlled with the RMS of 0.13 m
or smaller, and docking to an actual station was feasible.

3.2. Sampling Test at the Ocean
With a sampling system developed by Ishii andcoworkers of Kyushu Institute of Technology, TunaSand2 performed an experiment in which it collected seaJournal of Robotics and Mechatronics Vol.30 No.2, 2018

Underwater Platform for Intelligent Robotics and its Application
Enhanced image during observation

Enhanced image after returning

30

Touch down

Sea creature

20

about 0.5m

࣮Waypoint tracking
࣮Returning

Sea creature

Fig. 14. Comparison of images during observation and return.

0

-10

0.5

ReceivedcommandWest-east position [m]North-south position [m]

10

-20

-30

Sampling

-50
-40

-30

-20
-10
0
West-east position [m]1020

Fig. 13. Trajectory of the AUV in TS2-43 dive.

North-south position [m]-40Sampling
0.0

-0.5

-1.0
-33.0

-33.5

-34.0

-34.5
3.0

ber of the image containing a sampling target was designated and the AUV headed to the position where theimage was taken. For the AUV navigation, the horizontal
movement was controlled by designation of the positionsin the north-south and east-west directions in the UTM
coordinate system, the vertical movement was controlled
by designation of the altitude, and the rotation operation
was controlled by designation of the direction toward theimage shooting position. Fig. 14 shows an image captured
during the observation (in Mode 30) and another captured
when the AUV judged that it had returned to the point
where the former image was captured (in Mode 40). The
AUV considers that the two images in Fig. 14 were captured at the same position and therefore the precision of
the AUV’s position can be determined by comparing thetwo images. If we assume that the detected sea-creature in
the left image of Fig. 14 did not move until the time when
the right image of Fig. 14 was taken, we can conclude
that the AUV can perform self-localization with a precision of ±0.5 m or smaller. Fig. 15 shows the position of

Journal of Robotics and Mechatronics Vol.30 No.2, 20187

Altitude [m]creatures at a depth of 450–700 m offshore Hatsushima,
Shizuoka Prefecture (KS-17-J03 Research Cruise). The
sampling system, made by Ishii’s group, consists of alower-side camera, a manipulator with slurp gun [16–18],
and a control container. The sampling system removes
lighting irregularity from the captured seabed images [18]and uses a visual attention model [19] to find the highestattention image as living sea-creatures. The shooting position of the selected image is obtained from the statusserver and the compressed image is transmitted to thecontrol ship through an image-transmission acoustic modem on Tuna-Sand2 [20]. Researchers on the ship choosea sampling target from the received images and transmit the image number of the image containing the target
to the AUV, which then changes the mode to Mode 40.
After shifting to Mode 40, the AUV operates under thecontrol of the sampling system and returns to the position where the image of the designated image number
was taken. When the AUV reaches the shooting point,
the sampling system begins tracking the detected seacreatures (in Mode 50). When the relative error with thetarget becomes smaller than the threshold value or when
it performs tracking for a designated time, the manipulator with slurp gun performs sampling of the target (inMode 60).
We conducted a seabed image survey and sampling experiment with Tuna-Sand2 in the sea area offshore Hatsushima. Fig. 13 shows a trajectory of the AUV in TS-43
dive. After reaching the seabed, the AUV controlled andmaintained the altitude at 2.0 m and began navigation to
the waypoints with a speed of 0.12 m/s. In approximately
40 min after the start of the navigation, the image num-

2.0

1.0

0.0
0

100

200

300

400
500
time [s]

600

700

800

Fig. 15. Position of AUV during visual tracking by sampling system.

Nishida, Y. et al.

AUV while tracking after it returned to the shooting position. The figure indicates that the AUV controlled the horizontal position with the steady-state error of ±0.1 m or
smaller. The vertical movement under the altitude control
had larger tremors than the horizontal movement, with anovershoot of as large as 1.0 m. In the experiment, the sampling failed because of wrong recognition of the samplingtarget. However, the experiment showed that the AUV
could successfully accomplish the position control withhigh precision in the actual sea under the direction fromthe payload.

4. Conclusions
This paper explains the configuration of a hoveringtype AUV, which can navigate with the only the installed
sensors and the handling system, which receives operation commands from the payloads. The AUV uses a LAN
interface for the payloads, a status server that can acquire
the navigation information of the AUV, and a handlingserver that transmits target values from the payloads to the
AUV. The survey of the AUV was conducted sequentiallyby following a state flow, and the payloads worked in coordination with the AUV by executing processes according to the mode value received from the status server. The
modes that can be arranged for the payloads are available
in the state flow. In these modes, the payloads can safely
control the AUV’s position without checking the state of
the AUV.
To verify the effectiveness of the AUV and handlingsystem, a payload developed by Ishii and coworkers of
Kyushu Institute of Technology and one developed byMinami and coworkers of Okayama University was installed, and experiments with each of the payloads wereconducted. The experimental results showed that the
AUV and handling system worked correctly both in thewater tank and at sea, and the position of the AUV could
be controlled with a precision of ±0.1 m by using the payloads.
Acknowledgements
This work was supported by the Japan Science and Technology
CREST grant program for the “Establishment of core technology
for the preservation and regeneration of marine biodiversity andecosystems,” Grant Number JPMJCR11A2. The authors would
like to thank the research vessel Shinsei-maru crew and Japan
Agency for Marine-Earth Science and Technology (JAMSTEC)
during the KS-17-J03 research cruise.

References:

Surveys at 11,000 m Depth,” Marine Technology Society J., Vol.43,
No.5, pp. 87-96, 2009.
[4] S. Kawagucci, J. Miyazaki, R. Nakajima, T. Nozaki, Y. Takaya,
Y. Kato, T. Shibuya, U. Konno, Y. Nakaguchi, K. Hatada, H. Hiroyama, K. Fujikura, Y. Furushima, H. Yamamoto, T. Watsuji, J.
Ishibashi, and K. Takai, “Post-drilling Changes in Fluid Discharge
Pattern, Mineral Deposition, and Fluid Chemistry in the Iheya North
Hydrothermal Field, Okinawa Trough,” Geochemistry, Geophysics,
Geosystems, Vol.14, Issue 11, pp. 4774-4790, 2013.
[5] J. Hashimoto, S. Ohta, T. Gamo, H. Chiba, T. Yamaguchi, S.
Tsuchida, T. Okudaira, H. Watabe, T. Yamanaka, and M. Kitazawa,“First Hydrothermal Vent Communities from the Indian Ocean Discovered,” Zoological Science, Vol.18, No.5, pp. 717-721, 2001.
[6] M. Grasmueck, G. P. Eberli, D. A. Viggiano, T. Correa, G. Rathwell, and J. Luo, “Autonomous underwater vehicle (AUV) mapping
reveals coral mound distribution, morphology, and oceanography indeep water of the Straits of Florida,” Geophysical Research Letters,
Vol.33, Issue 23, L23616, 2006.
[7] M. Johnson-Roverson, O. Pizarro, S. B. Williams, and I. Mahon, “Generation and visualization of large-scale three-dimensional
reconstructions from underwater robotics surveys,” J. of Field
Robotics, Vol.27, Issue 1, pp. 21-51, 2009.
[8] B. Thorton, A. Bodenmann, O. Pizarro, S. B. Williams, A. Friedman, R. Nakajima, K. Takai, K. Motoki, T. Watsuji, H. Hirayama, Y.
Matsui, H. Watanabe, and T. Ura, “Biometric assessment of deepsea vent megabenthic communities using multi-resolution 3D image reconstructions,” Deep Sea Research Part I: Oceanographic Research Papers, Vol.116, pp. 200-219, 2016.
[9] Y. Nishida, T. Sonoda, S. Yasukawa, J. Ahn, K. Nagano, K.
Ishii, and T. Ura, “Development of an autonomous underwater vehicle with human-aware robot navigation,” Procs. of IEEE/OTS
OCEANS, 16506112, Monterey, 2016.
[10] Y. Nishida, T. Ura, T. Nakatani, T. Sakamaki, J. Kojima, Y. Itoh, andK. Kim, “Autonomous Underwater Vehicle “Tuna-Sand” for Image
Observation of the Seafloor at a Low Altitude,” J. of Robotics andMechatronics, Vol.26, No.4, pp. 519-521, 2014.
[11] Y. Nishida, T. Ura, T. Hamatsu, K. Nagahashi, S. Inaba, andT. Nakatani, “Resource Investigation for Kichiji Rockfish by Autonomous Underwater Vehicle in Kitami-Yamato Bank off Northern
Japan,” ROBOMECH J., Vol.1, No.2, pp. 1-6, 2014.
[12] A. Bodenmann, B. Thornton, and T. Ura, “3D Mapping of the
Seafloor in Color Using a Single Camera: Benthicmapping Basedon Video Recordings and Laser Profiling to Generate Colored 3D
Reconstructions of the Seafloor,” Sea Technol., Vol.51, No.12,
pp. 51-53, 2012.
[13] B. Thornton, A. Asada, A.Bodenmann, M. Sagekar, and T. Ura,“Instruments and Methods for Acoustic and Visual Survey of Manganese Crusts,” J. of Oceanic Engineering, Vol.38, No.1, pp. 186203, 2013.
[14] W. Song, F. Yu, and M. Minami, “3D visual servoing by feedforward evolutionary recognition,” J. of Advanced Mechanical Design,
Systems, and Manufacturing, Vol.4, No.4, pp. 739-755, 2010.
[15] F. Yu, M. Minami, W. Song, and A. Yanou, “Eye-vergence visual
servoing enhancing Lyapunov-stable trackability,” Artificial Lifeand Robotics, Vol.18, No.1-2, pp. 27-35, 2013.
[16] T. Sonoda, A. A. F. Nassiraei, I. Godler, T. Weerakoon, and K. Ishii,“Development of Hydraulic Underwater Manipulator for Deep-sea
Survey AUV,” Procs. of ICRAOB, pp. 242-245, 2017.
[17] T. Weerakoon, T. Sonoda, A. A. F. Nassiraei, I. Godler, and K. Ishii,“Underwater Manipulator for Sampling Mission with AUV in DeeSea,” Procs. of JSME Conf. on Robotics and Mechatronics, 2P1F11, 2017.
[18] A. Jonghyun, S. Yasukawa, T. Sonoda, T. Ura, and K. Ishii, “Erratum to: Enhancement of Deep-sea Floor Image Obtained by an
Underwater Vehicle and Its Evaluation by Crab Recognition,” J. of
Marine Science and Technology, Vol.22, No.4, pp. 758-770, 2017.
[19] L. Itti and C. Koch, “A saliency-based search mechanism for overt
and covert shifts of visual attention,” Vision research, Vol.40,
No.10, pp. 1489-1506.
[20] A. Jonghyun, S. Yasukawa, T. Sonoda, Y. Nishida, K. Kazuo, and
T. Ura, “Image Enhancement and Compression of Deep-sea Floor
Image for Acoustic Transmission,” Proc. of MTS/IEEE OCEANS,
CFP16OCF-ART, 2016.

[1] C. E. Fitzgerald and K. M. Gillis, “Hydrothermal Manganese Oxide Deposits from Baby Bare Seamount in the Northeast Pacific
Ocean,” Marine Geology, Vol.225, Issues 1-4, pp. 145-156, 2006.
[2] S. Thatje, S. Hall, C. Hauton, C. Held, and P. Tyler, “Encounterof Lithodid Crab Paralomis Birsteini on the Continental Slope
off Antarctica,” Sampled by ROV, Polar Biology, Vol.31, Issue 9,
pp. 1143-1148, 2008.
[3] H. Yoshida, S. Ishibashi, Y. Watanabe, T. Inoue, J. Tahara, T. Sawa,
and H. Osawa, “The ABISMO Mud and Water Sampling ROV for

8

Journal of Robotics and Mechatronics Vol.30 No.2, 2018

Underwater Platform for Intelligent Robotics and its Application

Name:

Name:

Yuya Nishida

Shinsuke Yasukawa

Affiliation:

Affiliation:

Project Assistant Professor, Frontier Research
Academy for Young Researchers, Kyushu Institute of Technology

The University of Tokyo
Recreation Lab, Inc.

Address:

Address:

2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Tokyo 808-0196, Japan

4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan
4F Otemachi Bldg., 1-6-1 Otemachi, Chiyoda-ku, Tokyo 100-0004, Japan

Brief Biographical History:
2011- Researcher, Kyushu Institute of Technology
2012- Project Researcher, Institute of Industrial Science, The University ofTokyo
2015- Project Assistant Professor, Kyushu Institute of Technology

Main Works:• “Design Principle of High Power Joint Mechanism Possible to Walking
and Jumping Imitating Locust Leg Structure,” J. of Robotics andMechatronics, Vol.23, No.2, pp. 225-230, 2011.
• “Resource Investigation for Kichiji Rockfish by Autonomous
Underwater Vehicle in Kitami-Yamato Bank off Northern Japan,”
ROBOMECH J., Vol.1, Issue 1:2, pp. 1-6, 2014.

Membership in Academic Societies:• The Japan Society of Mechanical Engineers (JSME)• The Robotics Society of Japan (RSJ)• The Japanese Society of Fisheries Science

Brief Biographical History:
2014-2017 Research Associate, Kyushu Institute of Technology
2017 Received Ph.D., Division of Electrical, Electronic and Information
Engineering, Osaka University
2017- Researcher, Recreation Lab, Inc.
2017- Research Associate, The University of Tokyo

Main Works:• “A Vision Sensor System with a Real-Time Multi-Scale Filtering
Function,” Int. J. of Mechatronics and Automation, Vol.4, No.4,
pp. 248-258, 2014.
• “Real-Time Object Tracking Based on Scale-Invariant Features
Employing Bio-Inspired Hardware,” Neural Networks, Vol.81, pp. 29-38,
2016.

Membership in Academic Societies:• The Robotics Society of Japan (RSJ)• The Society of Instrument and Control Engineers (SICE)

Name:
Name:

Takashi Sonoda

Kazunori Nagano

Affiliation:
Affiliation:

Kyushu Institute of Technology

Project Researcher, Institute of Industrial Science, The University of Tokyo

Address:
2-4 Hibikino, Wakamatsu-ku, Kitakyushu 808-0196, Japan

Address:

Brief Biographical History:

4-6-1 Komaba, Meguro-ku, Tokyo 153-8505, Japan

2009- Researcher, Fukuoka Industry, Science and Technology Foundation
2012- Researcher, Kyushu Institute of Technology
2015- Research Associate Professor, Kyushu Institute of Technology

Brief Biographical History:

Main Works:

2013- Joined Mitsui Engineering and Shipbuilding Co., Ltd.
2014- Project Researcher, Institute of Industrial Science, The University ofTokyo• “Development of antagonistic wire-driven joint employing kinematictransmission mechanism,” J. of Automation, Mobile Robotics & Intelligent
Systems, Vol.4, No.2, pp. 62-70, 2010.
• “Modeling and Evaluation of a Twist Drive Actuator for Soft Robotics,”
Advanced Robotics, Vol.26, No.7, pp. 765-783, 2012.

Membership in Academic Societies:• The Robotics Society of Japan (RSJ)• The Society of Instrument and Control Engineers (SICE)

Journal of Robotics and Mechatronics Vol.30 No.2, 20189

Nishida, Y. et al.

Name:

Name:

Mamoru Minami

Tamaki Ura

Affiliation:

Affiliation:

Professor, Graduate School of Natural Scienceand Technology, Okayama University

Center for Socio-Robotic Synthesis, Kyushu Institute of Technology

Address:

Address:

3-1-1 Tsushimanaka, Okayama 700-8530, Japan

2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka 808-0196, Japan

Brief Biographical History:

Brief Biographical History:

1994- Associate Professor, Department of Mechanical Engineering,
University of Fukui
2002- Professor, Department of Human and Artificial Intelligence
Systems, University of Fukui
2010- Professor, Graduate School of Natural Science and Technology,
Okayama University

1992-2013 Professor, The University of Tokyo
2013- Professor, Kyushu Institute of Technology

Main Works:• “Dual-eyes Vision-based Docking System for Autonomous Underwater
Vehicle: An Approach and Experiments,” J. of Intelligent & Robotic
Systems, Doi: 10.1007/s10846-017-0703-6, 2017.
• “Visual Servoing to catch fish Using Global/local GA Search,”
IEEE/ASME Trans. on Mechatronics, Vol.10, No.3, pp. 352-357, 2005.
• “Reconfiguration Manipulability Analyses for Redundant Robots,”
Trans. of the ASME, J. of Mechanisms and Robotics, Vol.5, No.4,
pp. 1-16, 2013.

Main Works:• Development and operation of AUV “r2D4”

Membership in Academic Societies:• The Institute of Electrical and Electronics Engineers (IEEE) Oceanic
Engineering Society (OES), Fellow• The Robotics Society of Japan (RSJ)• The Japan Society of Naval Architects and Ocean Engineers

Membership in Academic Societies:• The Institute of Electrical and Electronics Engineers (IEEE)• The Japan Society of Mechanical Engineers (JSME)• The Society of Instrument and Control Engineers (SICE)• The Robotics Society of Japan (RSJ)

Name:
Kazuo Ishii

Affiliation:
Professor, Department of Human Intelligence
Systems, Kyushu Institute of Technology

Address:
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka 808-0196, Japan

Brief Biographical History:
1996- Researcher, Institute of Industrial Science, The University of Tokyo
1996-1998 Assistant Professor, Kyushu Institute of Technology
1998-2011 Associate Professor, Kyushu Institute of Technology
2011- Professor, Kyushu Institute of Technology

Main Works:• “Enhancement of deep-sea floor images obtained by an underwater
vehicle and its evaluation by crab recognition,” J. of Marine Science andTechnology, Vol.22, Issue 4, pp. 758-770, 2017.

Membership in Academic Societies:• The Institute of Electrical and Electronics Engineers (IEEE)• The Japan Society of Mechanical Engineers (JSME)• The Robotics Society of Japan (RSJ)• The Institute of Electrical and Electronics Engineers (IEEE)

10

Journal of Robotics and Mechatronics Vol.30 No.2, 2018