# High-Speed Underwater Optical Wireless Communication with Advanced Signal Processing Methods Survey
## 4 Machine Learning Applications in UOWC
### Reinforcement Learning in UOWC Systems  
Reinforcement learning is often used in UOWC systems to improve the connection success rate of the underwater sensor network and AUVs, further increasing the reliability of links. In underwater sensor networks, the highly dynamic topology can hinder the routing of UOWC links, particularly due to the ocean flow movement. In [121], an advanced routing protocol based on multi-agent reinforcement learning (MARL) was proposed to increase the link reliability and communication quality in the UOWC-based sensor network. Simulation results show that the UOWC-based sensor network with MARL has 80% average residual energy of the network after 150 simulation times compared with 30% and 70% achieved with the Q-learning-based delay tolerant routing (QDTR) and the ad hoc on-demand distance vector (AODV) routing protocols. Moreover, MARL has the highest delivery ratio in the static network (95.87%) and the dynamic network (95.57%) compared with the other two schemes (49.5% and 44.56% achieved in AODV in the static network and the dynamic network, respectively). In addition, to overcome the same limitations, an efficient routing protocol based on MARL was also employed in another work [122]. Simulation results show that the MARL routing protocol provided the same low power consumption advantages and high-quality UOWC linksin a network with 14 neighboring nodes.  
Moreover, reinforcement learning algorithms have also been exploited to solve pointing acquisition and tracking (PAT) problems between underwater applications (AUVs). An advanced beam adaptation method based on the state-action-reward-state-action (SARSA) algorithm for point-to-point UOWC systems was proposed in [118]. Results show that the SARSA-based beam adaptation method increased the success rate from 66% to 93%, which further achieved better link reliability. Moreover, the SARSA increased SNR by 6 to 10 dB compared to the traditional NML method in different types of underwater channels. Similarly, to overcome the poor link reliability and to optimize the connecting success rates between two AUVs, the soft actor-critic reinforcement learning algorithm was designed in [119]. Results show that the success rate for the transmitting AUV to maintain the LOS link for ten time steps was 97.53% after 10,000 episodes in the simulation environment. In [120], a deep reinforcement learning algorithm assisted by an extended Kalman filter was employed to improve the reliability of water-air optical wireless communication between AUVs and unmanned aerial vehicles (UAVs), which is even more challenging compared with connecting AUVs only. Results show that the proposed learning algorithm achieves a shorter MSE (0.02 m) compared with the triangular exploration (TE) algorithm (0.06 m), a shorter flight distance (1.1 m compared to 2 m on TE), and a smoother trajectory (3.23 compared to 6.98 on TE), which implies a higher alignment accuracy and smaller energy consumption. Moreover, it also improves the link availability by 25% compared with the TE algorithm.