Visual Aided Deep Learning Applications for Underwater Operations

The 4 industrial revolution has been ongoing the last four years and results are starting to show in the market. Google constantly reveals new technology that can beat humans at different strategy games and they have developed AIs that can learn to walk on their own. Also, governments and classification companies continuously works towards a set of regulations and guidelines for autonomous ships, Yara Birkeland is a ship that is planned to be released in 2020 that will move towards fully autonomous operations within 2022. Autonomy in underwater operations are also experiencing a growth which shows in the expected market growth. Machine learning is believed to play an important role in the shift towards autonomy. Feature extraction methods for subsea applications existing today mainly consist of acoustic sensors. Feature extraction using camera vision are rarely used, but improvements within neural networks, especially convolutional neural networks (CNN), have proved promising results. This thesis will investigate the use of such methods for subsea applications. The thesis specifically highlights two different objectives, that are dispersed at two appended articles. The first objective concerns the reality gap, which describes the phenomena that occurs when transferring knowledge between domains, specifically the artificial simulated domain and the real world domain. Existing methods focusing on this have been explored and compared in order to generate an outline of existing solutions. Due to the harsh environment in subsea applications, the use of camera vision for feature extraction has been shallow. Bad lighting, marine snow, constantly change of environment etc. makes if difficult to model the subsea environment. Thus, there will almost certainly exist a gap between a simulated environment and the real world. In the last few years numerous frameworks targeting domain transfer have been published. Generative Adversarial Networks (GAN) has stood out as maybe one of the recent most promising methods regarding domain transfer. This thesis investigate the use of a framework called CycleGAN which introduces a cycle consistent feature in the ordinary GAN. CycleGAN has proven good results in several disciplines and has in this thesis been tested for the first time for datasets regarding an underwater environment. The method consist of taking an image as input and generating a similar image in another domain, e.g. taking an image of a rendered subsea panel as input and generating a real-looking subsea panel in a real underwater environment as output. In this thesis the framework is applied on two different underwater image datasets and is proven to accomplish a good mapping between simulated and real domains. The second objective concerns visual based object detection using monocular camera. Deep learning methods for object detection are investigated in collaboration with methods for extracting relevant features from the respective detectors in order to aid a dynamic positioning (DP) system. A detector based on YOLOv3 is applied on a dataset collected from the Marine Cybernetics Laboratory at NTNU. The dataset includes a known object in the laboratory pool environment. Once the detector is trained and successfully detects the relevant object, a spatial scaling function and a DP system for the underwater vehicle BlueROV2 is designed. The BlueROV2 performs DP relative to the object extracting position information and spatial features from the detector. The system performs well and the BlueROV2 manages to perform DP in a good manner with correct extraction of both localization and spatial features of the object of interest.