Proceedings of the Thirty-second (2022) International Ocean and Polar Engineering Conference
Shanghai, China, June 5–10, 2022
Copyright © 2022 by the International Society of Offshore and Polar Engineers (ISOPE)
ISBN 978-1-880653-81-4; ISSN 1098-6189

www.isope.org

Reinforcement Learning for Maneuver Control of a Bio-Inspired Vessel with Undulating Fin Propulsion
Gonzalo Garcia, Mohammad Uddin, Siddhartha Verma, and Oscar Curet
Department of Ocean and Mechanical Engineering, Florida Atlantic University
Dania Beach, Florida, United States

device equipped with an undulatory fin actuated by 16 rays, with thecapability of controlling each ray asynchronously, regulating its positionand speed for arbitrary path following. The degree of intricacy of thesystem, while trying to keep a close resemblance to the natural swimmer,
strongly suggests the need for a machine learning approach, especially
technique such as reinforcement learning that is capable of learning
during operation. We have developed a two-input two-output 3 DOF
path-following control system for a bio-inspired robot based on aspecific pattern of ray motion. The controller modulates the amplitude ofa traveling wave for speed control, and coordinates the offsetting of asubset of rays for the lateral-directional dynamics, while using the closest
distance to the trajectory and its current speed as inputs. The
reinforcement learning controller was tested in various conditions
experimentally, including an indoor flume, and indoor and outdoor open
tanks to perform different maneuvers, with different levels of external
disturbances.

ABSTRACT
Undulating fins provide biological swimmers with a remarkablepropulsion system. Replication with autonomous underwater robots is
not simple, both on the technical side and due to the complexity of thecontrol system. Reinforcement learning is applied to a bio-inspired
vessel for maneuver control by manipulating the kinematics of its
undulating membrane. The controller is initially trained with a numericalmodel recreating the distributed forces and moments on the fin, and thenapplied to an underwater vehicle. It was found that using Reinforcement
Learning the vehicle was able to perform key maneuvers including
control for speed, heading, and turning. The results demonstrate thatreinforcement learning has the potential to overcome high levels ofsystem complexity while delivering an optimal solution.

KEY WORDS: Reinforcement learning; DDPG; undulating finpropulsion.

REINFORCEMENT LEARNINGINTRODUCTION

The level of complexity in current autonomous vehicles makes classicalcontrol approaches ineffective, or harder to design, due to an increasing
number of sensors, actuators, control loops, and involved dynamics.
Reinforcement learning (RL) addresses these limitations in an integrated
way, providing an optimal solution independently of the complexity ofthe system under control.
The RL agent learns by interacting with the environment, which
normally not only includes the system’s dynamics, but environmental
perturbations as well. The agent learns actively while operating in realtime. This is in contrast to other optimal methods that base their learningon simplified mathematical models and are best designed to be tuned
backward-in-time (Sutton and Barton, 2018; Busoniu, Babuska, De
Schutter, and Ernst, 2017).
The specific RL version chosen for this work is known as Deep
Deterministic Policy Gradient (DDPG), which is an actor-critic RL agent
with a model-free, online, off-policy algorithm that maximizes theexpected cumulative long-term reward. This method allows for anoptimal learning process carried out during operation, based on therobot’s actual dynamics, and continuous-time signals. In the limit, thecritic section captures the optimal action-state combination in terms of
the highest long-term reward, and the actor section implements theoptimal control policy.

Biological swimmers have achieved remarkable propulsion capabilities
through undulatory fins. Nature has evolved to a point where the use ofa single fin in combination with pectoral fins are able to controlswimming in multiple directions with high levels of accuracy. The bioinspired application of these traits to underwater vessel locomotion
presents a non-trivial task. The bio-inspiration was taken from theGymnotiformes, known as South American knifefish. Normally, the fish
has an elongated fin along its body, driven by hundreds of bones in an
undulatory manner, producing the necessary forces and moments tomove freely in all directions. It has been observed that these kinds of fish
have developed several fin kinematics for different kinds of motion,
ranging from sinusoidal traveling wave-like kinematics for longitudinalmotion, to combination of more than one wave for vertical displacement
among others, all combined with body bending for lateral-directional
movement. The potential artificial mimicking by underwater robotic
devices imposes a level of complexity in the design of control systems
that quickly rules out traditional linear model-based controllers. The
operation of a robotic undulatory fin composed of several rotatory raysin place of the fish fin bones, entails a multivariable process that requires
a different approach. Our current research involves an autonomous

1072