# Shear Wave Velocity Estimation Based on Deep-Q Network
## 5 Numerical Experiments
### Case 1  
Case 1 defines a six-layer geoacoustic model referenced from the inversion results in the Grane field [26], as shown in Table 4. The phase velocity dispersion curves for the first five modes of the Scholte waves are calculated in the frequency range from 0 to 5 Hz as the ground truth. \begin{table}
\begin{tabular}{c c} \hline
**Item** & **Value** \\ \hline Mini-batch size & 32 \\ Memory size & 100,000 \\ Update frequency & 4 \\ Discount factor & 0.9 \\ Learning rate & 0.0001 \\ Factor in greedy strategy & 0.9 \\ Training epoch & 300 \\ \hline \end{tabular}
\end{table} 
Table 2: Hyperparameters of the DQN framework. \begin{table}
\begin{tabular}{c c c} \hline
**Algorithm** & **Description** & **Value** \\ \hline \multirow{4}{*}{All} & Maximum of iteration (\(i_{max}\)) & 100 \\  & Number of sampling per iteration (\(k\)) & 200 \\  & Threshold of misfit (\(e_{\text{threshold}}\)) & 10 (m/s) \\  & Threshold of convergence (\(e_{\text{convergence}}\)) & 0.1 \\ \multirow{2}{*}{GA and DE} & Mutation rate & 0.001 \\  & Population size & 200 \\ \multirow{2}{*}{ASSA} & Temperature reduction factor & 0.995 \\ \hline \end{tabular}
\end{table} 
Table 3: Preset parameters of the candidate methods. The density and compression wave velocities are considered as known since they are not sensitive to the dispersion property of the Scholte wave [1]. The original search bounds for Case 1 are shown in Table 5.  
The estimated dispersion curves with the ground truth are shown in Figure 3, where the black dots are the ground truth. The blue, green, yellow, and red lines express the estimated dispersion curves by the GA, ASSA, DE, and DQN framework, respectively. The estimated shear wave velocity profiles with the ground truth are shown in Figure 4, where the black line refers to the ground truth. The blue, green, yellow, and red lines express the estimated shear wave velocity profiles by the GA, ASSA, DE, and DQN framework, respectively. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Layer** & \(\mathbf{h}\) **(m)** & \(\mathbf{\rho}\) **(kg/m\({}^{3}\))** & \(\mathbf{Vp}\) **(m/s)** & \(\mathbf{Vs}\) **(m/s)** \\ \hline
0 (Ocean) & 125 & 1.0 & 1490 & 0 \\
1 & 35 & 1.5 & 1750 & 365 \\
2 & 89 & 1.8 & 2000 & 696 \\
3 & 125 & 2.0 & 3500 & 878 \\
4 & 224 & 2.2 & 3750 & 1060 \\
5 (Half-space) & 956 & 2.1 & 3250 & 1140 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Geoacoustic model for Case 1. Figure 3: The estimated dispersion curves with the ground truth in Case 1. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer** & \(\mathbf{h}\) **(m)** & \(\mathbf{Vs}\) **(m/s)** \\ \hline
0 (Ocean) & - & - \\
1 & \([10,70]\) & \([100,500]\) \\
2 & \([40,120]\) & \([250,850]\) \\
3 & \([50,350]\) & \([500,1200]\) \\
4 & \([50,650]\) & \([800,1500]\) \\
5 (Half-space) & - & \([700,1800]\) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Search bounds for Case 1. The performance analysis for Case 1 is shown in Table 6, in which the bold fonts refer to the lowest relative error, misfit value, or running time over each row. The following phenomena are revealed based on Figures 3 and 4, and Table 6:  
* At the low frequency of the fundamental mode, the estimated errors by ASSA and DE are larger than that by the GA and DQN framework. * For all candidate methods, the relative errors of the shear wave velocity in all layers are lower than that of thickness. Both the shear wave velocity and layer thickness are sensitive to the dispersion properties of Scholte waves. However, in some cases, one can be more sensitive than the other. * The low-to-high ranking for misfit values is the DQN framework, ASSA, DE, and GA. It is the same as the low-to-high ranking for the relative error of each geoacoustic parameter with a few exceptions of \(h_{1}\), \(h_{2}\), \(h_{4}\), and \(Vs_{2}\). This illustrates that the misfit value has a partly positive correlation with the overall relative errors, which is significant since the ground truth of a field survey is mostly unknown and the misfit value is the only metric.