# Shear Wave Velocity Estimation Based on Deep-Q Network
## 1 Introduction  
Shear wave velocity estimation is an important geoacoustic inversion task for seabed geotechnical applications since shear wave velocity can provide a good indicator of sediment rigidity and characterization [1; 2]. The seabed shear wave velocity profile can be estimated from the dispersion curve of the seismoacoustic interface waves, which is a convenient and low-cost approach compared to the direct approach (e.g., coring). Here, the interface waves refer to Scholte waves since in most underwater and seismic experiments sources are deployed in the water column and only Scholte waves can be generated [2]. There are two approaches for geoacoustic inversion [3]: the optimization-based approach and the machine learning (ML)-based approach. The optimization-based approach exploits the optimization method for determining a set of geoacoustic parameters that best fit the measured data. Based on the previous reviews [4; 5], some optimization methods have been demonstrated to perform well for geoacoustic inversions, such as the genetic algorithm (GA) [6], differential evolution (DE) [7], and adaptive simplex simulated annealing (ASSA) [8]. On the other hand, with the development of ML, studies for geoacoustic inversion based on ML have appeared. Most of the studies are based on supervised learning, which aims to train a deep neural network for inversion based on a vast dataset [9; 10; 11; 12]. This type of approach normally consists of the following steps: (1) creating a simulation dataset based on a physical forward model; (2) training a deep neural network based on the simulation dataset; (3) exploiting the trained neural network for the real-world inversion. These two approaches can both provide acceptable performances for geoacoustic inversion. However, they also have some drawbacks. Since the prevalent optimization methods are not specifically designed for geoacoustic inversion, they may incur some limitations and difficulties to be applied in this specific field, such as difficulties in choosing the hyperparameters required by the algorithm, which may incur more time-costs. The ML-based approach introduces a drawback that the trained neural network cannot interactwith the physical forward model. The procedure of creating the simulation dataset has to be repeated when the ocean environment changes significantly. To avoid the drawbacks and keep the interactive ability of the optimization-based approach and the learnability of the ML-based approach, deep reinforcement learning (DRL) could become a potential option. Unlike supervised learning, DRL learns by trial and error, which iteratively updates the model by interacting with the environment to achieve good data fitting [13]. Its potential for geoacoustic inversion can be intuitively interpreted. The physical forward model role, e.g., the environment, can create a replica according to a set of geoacoustic parameters input by the DRL model. The DRL method can update the model by iteratively modifying the replica to obtain the best fit of the measured data. DRL has been widely used as an intelligent controller for different purposes, including robotics [14], electronic sports [15], automatic controlling [16; 17; 18; 19; 20; 21; 22; 23], etc. Specifically, [PERSON] et al. demonstrated the effectiveness of DRL for controlling physical robots [14]. [PERSON] et al. proposed a green signal time allocation system based on a deep Q-network (DQN) for reducing the standard deviation of each lane at an intersection [16]. [PERSON] et al. modeled penetration testing as a Markov decision process and exploited DQN for autonomous penetration testing [17]. [PERSON] et al. exploited a DRL-based DQN agent for a visual object-tracking task in a virtual environment. It has been demonstrated that the proposed agent outperforms some conventional methods of two public databases [18]. [PERSON] et al. proposed a DRL-based method to solve the relay selection problem in the decode and forward relay-aided free-space optical communication system [19]. [PERSON] et al.