## 4 PER-SAC based Optimal Parameter Autotuning for DBSCAN  
### PER-SAC Algorithm  
The PER-SAC algorithm combines the SAC algorithm with the PER mechanism to fast search the stabilized optimal parameter for DBSCAN. The SAC algorithm is a DRL algorithm of the actor-critic type considering maximum entropy, which increases the exploration degree of the DRL model by adopting a more stochastic strategy, exhibiting excellent performance on both discrete action tasks and continuous action tasks. Figure 7: Illustration of spatial-temporal random partition. The experience playback mechanism has been used in many DRL algorithms such as TD3 and SAC. However, different experiences do not have the same value to the DRL agent and have different impacts on the strategies. But the uniform random sampling ignores, to some extent, those experiences with high values that can better optimize the strategies. Based thereon, this paper adopts the prioritized experience playback mechanism on the basis of uniform sampling of the experience replay buffer of the SAC algorithm to construct a DRL framework for the parameter autotuning of DBSCAN. The PER mechanism first uses the value of the time-difference deviation (TD-error) of each experience, i.e., the absolute value of the difference between the Target_Q value and the Current_Q value, as an index to evaluate the value of the experience. The larger the absolute value of the TD-error, the more important the experience to be learned by the agent. The experiences in the experience replay buffer are sorted in order of their value. Multiple sampling and playback are possible for certain high-value experiences. In addition, the bias problem due to changes in the sample distribution is solved by the importance sampling weight (ISW). In the preliminary \"warm-up\" phase of the algorithm, the agent adopts an exploration strategy to interact with the environment, obtaining random samples and storing them in the experience buffer until enough samples have been stored in the experience buffer. When starting training, the PER-SAC algorithm first samples according to the weights and draws _batch_size_ size samples from the prioritized experience replay buffer. The algorithm calculates the TD-error of each sample separately based on the drawn samples and obtains the loss values of each network based on the individual network loss function \(J_{Q}(\theta)\), \(J_{\pi}(\varphi)\), \(J(\alpha)\). The loss functions of Q-network \(Q(\theta)\), the strategy \(\pi(\varphi)\), and the weighting factor \(\alpha\) are shown in Equations (2)-(4), respectively, where \(E\) is the expectation function and \(\overset{-}{H}\) is the minimum expected entropy of the expectation. The parameter explanation of Equations (2)-(4) is listed in Table 1.  
\[ \begin{array}{c}J_{Q}(\theta)=E_{(s_{t},a_{t})\sim D}[\frac{1}{2}w_{t}(Q_{ \theta}(s_{t},a_{t})-(r(s_{t},a_{t})+\\ E_{s_{t+1}\sim p}(s_{t+1}|s_{t},a_{t})\left[\gamma\pi_{t}(s_{t+1})^{T}(Q_{ \overline{\theta}}(s_{t+1},a_{t+1})-\alpha log(\pi_{\varphi}(a_{t+1}|s_{t+1}) ))\right])^{2}]\\ \\ J_{\pi}(\varphi)=E_{s_{t}\sim D}[w_{t}\pi_{t}(s_{t})^{T}[\alpha log(\pi_{ \varphi}(s_{t}))-Q_{\theta}(s_{t})]]\\ \\ J(\alpha)=w_{t}\pi_{t}(s_{t})^{T}[-\alpha(log(\pi_{\varphi}(s_{t}))+\overline{ H})]\\ \\ \end{array}  \tag{2}\]  
Each network updates its corresponding network parameters according to the above loss functions. When all the samples in a _batch_size_ are trained, the PER-SAC algorithm updates the latest priorities to the corresponding samples in the prioritized experience replay buffer. The DRL network structure based on PER-SAC is shown in Figure 8.