## 4 PER-SAC based Optimal Parameter Autotuning for DBSCAN
### PER-SAC Algorithm
\begin{table}
\begin{tabular}{c c} \hline \hline
**Parameters** & **Explanation** \\ \hline \(s_{t}\) & State of the environment at the moment \(t\) \\ \(a_{t}\) & The action performed by the agent at the moment \(t\) \\ \(r_{t}\) & Feedback on the action \(a_{t}\) performed by the agent in the state \(s_{t}\) of the moment \(t\) \\ \(\theta\) & Q network parameter \\ \(\varphi\) & Policy network parameter \\ \(\alpha\) & Weighting factor \\ \(w\) & Importance sampling weight \\ \(\gamma\) & Discount factor \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Parameter explanation of equations. The pseudo-code of the PER-SAC algorithm is shown in Algorithm 3. ```
Input: Q-network parameters \(\theta_{1}\), \(\theta_{2}\), policy-network parameter \(\varphi\) Output: \(\theta_{1}^{*}\), \(\theta_{1}^{*}\), \(\varphi^{*}\)
1 Initialize Q-network parameters \(\theta_{1}\), \(\theta_{2}\), policy-network parameter
2 Initialize Q-network parameters
3\(\overline{\theta_{1}}=\theta_{1},\ \overline{\theta_{2}}\leftarrow\theta_{2}\)
4//Initializing an empty PER experience replay buffer \(D\)
5\(D\gets D\)
6 foreachiterationdo
7 foreachenvironment step \(k=1\ldots T\)do
8\(a_{t}\sim\pi_{\varphi}(a_{t}|s_{t})\) //Sample action from the policy
9\(s_{t+1}\sim p(s_{t+1}|s_{t},a_{t})\)//Sample transition from the environment
10\(D\gets D\cup\left\{s_{t},a_{t},r(s_{t},a_{t},s_{t+1}),s_{t+1}\right\}/\)Store the transition in the replay buffer
11 endfor
12 if\(D\).capacity > warmup sizethen
13 for eachgradient stepdo
14//Sample samples \(S\), sample indices indices, weights \(w\) from \(D\)
15\(S\), indices, \(w\sim D\)
16\(I_{Q}=I_{Q}^{S}\)//Loss function for the Q-network
17\(J_{\pi}=I_{\overline{\theta}}^{S}\)//Loss function for the policy network
18\(I_{a}=I_{\overline{\theta}}^{S}\)//Loss function for the weighting factor \(\alpha\)
19\(\theta_{i}\leftarrow\theta_{i}-w\lambda_{Q}\ \
abla_{\theta_{i}}J_{Q}(\theta_{i})\quad i\in\{1,2\}\) //Update the Q-network parameters
20\(\varphi\leftarrow\varphi-w\lambda_{\overline{\
abla}}\varphi_{J}\ \pi_{\overline{\varphi}}(\varphi)\ /\)Update the policy network parameters
21\(\alpha\leftarrow\alpha-w\lambda\
abla_{a}\ I_{a}\ /\)//Adjust temperature factor
22\(\overline{\theta_{i}}\leftarrow\tau\theta_{i}+(1-\tau)\overline{\theta_{i}} \quad i\in\{1,2\}\) //Update Target network weights
23\(p\leftarrow\mid\!\!\!\!\! eratively autotuned by training the agent and interacting with the environment until the optimal parameters are obtained. The agent regards the parameter space and the DBSCAN algorithm as the environment, the result obtained by the clustering algorithm as the state, and the direction of parameter tuning as the action, and it performs iterative optimization. In addition, we use a small number of samples (i.e., 20% of the original dataset) to reward the agent with superior performance in a weakly-supervised manner and optimize the policy functions of the agent based on the PER-SAC framework. The overall structure of the optimal parameter autotuning framework based on the PER-SAC algorithm in a single episode is shown in Figure 9. The actor network selects the action \(a_{t}\) to be executed based on the current state \(s_{t}\). The state of the environment is transformed \(s_{t+1}\) to after executing the action \(a_{t}\), and inputs (\(s_{t}\), \(a_{t}\)) into the critic network for scoring to calculate the reward \(r_{t}\). Then, the current experience (\(s_{t}\), \(a_{t}\), \(s_{t+1}\), \(r_{t}\)) is stored into the PER buffer. During subsequent training, experiences are selected and learned from this buffer based on prioritization for updating the parameters of the network and tuning the parameter set of {_Eps_, _MinPts_}. The framework performs in this manner until reaching the parameter boundaries or the upper limit of the number of searches, and the search process for this episode ends. For the reward function used in the parameter autotuning framework, we use 20% of samples to provide an external measure of the effect of clustering and as a basis for reward.