# Adaptive Clutter Intelligent Suppression Method Based on Deep Reinforcement Learning
## 4 Adaptive Clutter Intelligent Suppression Method Based on DRL
### Reinforcement Learning Element Design  
(1) State-Space Design: In this paper, the state space is defined as the set of features observed by an agent in the environment, which is the echo data of the \(l\)th range cell in which the agent is located. \[S=[x_{1,l},x_{2,l},\dots x_{N,l}] \tag{9}\]  
where \(N\) represents the number of pulses accumulated, and \(l\) represents the range cell (\(1\leq l\leq L\)) in which the agent is located. The environmental state space is reset after each episode. (2) Action-Space Design: The action space is the set of possible actions that an agent can take, i.e., the range of filter parameters that can be adjusted. Specifically, the action  
Figure 2: The network parameter update process of DQN. space includes the adjustment of three parameters of the filter: filter order, spectral width, and spectral center. In this paper, the action space is designed as \(A\), as follows:  
\[A=[f_{0}+0.1,f_{0}-0.1,\delta_{f}+0.1,\delta_{f}-0.1,M+1,M-1] \tag{10}\]  
where \(M\) is the filter order, taking the value range as (\(3\leq M\leq 16\)), \(f_{0}\) is the spectral center, with the value range of (\(-PRF/2\leq f_{0}\leq PRF/2\)), and \(\delta_{f}\) is the spectral width, with the value range of \(0.1\leq\delta_{f}\leq 120\). After the filter initial module, the agent is able to gradually adjust the parameters of the filter during the training process so that it gradually approaches the optimal solution. This iterative parameter update strategy can maintain the stability of the parameters while dynamically adjusting the filter according to the different observation data in the environment. The filter parameters of each agent are designed according to the range cell in which it is located, which means that each agent has its own unique filter parameters, and this filter design can better adapt to the environmental information of different range cells, so as to achieve the targeted suppression of the clutter in this cell. (3) Reward function design: In this paper, the SCNR of the echo data matrix and the clutter attenuation (CA) of individual range cells are used as the evaluation metrics. The former directly reflects the improvement of the overall performance, and the latter takes into account the clutter attenuation of individual range cells. By evaluating the behavior of the agent through the partial metric of CA, the agent is able to adjust the filter parameters more finely and optimize for the clutter situation at different range cells, which helps to improve the adaptability and generalization ability of the system. The reward function is defined as follows:  
\[SCNR=10\log_{10}(\frac{P_{s}}{P_{c}}) \tag{11}\]  
\[CA=\frac{C_{i}}{C_{o}} \tag{12}\]  
where \(P_{s}\) is the target average power of the range cell where the target is located, and \(P_{c}\) is the clutter average power (excluding the range cell where the target is located, the average power of the whole echo data matrix). In Equation (12), \(C_{i}\) is the power before the clutter suppression of a single range cell, and \(C_{o}\) is the power after clutter suppression. \[R=\left\{ \begin{array}{ll}100,&\text{if SCNR}_{t}{>}\text{ SCNR}_{\text{peak}}\text{ and CA}_{t}{>}\text{ CA}_{\text{peak}}\\ 30,&\text{if SCNR}_{t}\leq\text{ SCNR}_{\text{peak}}\text{ and CA}_{t}{>}\text{ CA}_{\text{peak}}\\ -1,&\text{if SCNR}_{t}{>}\text{ SCNR}_{\text{peak}}\text{ and CA}_{t}\leq\text{ CA}_{\text{peak}}\\ -10,&\text{if SCNR}_{t}\leq\text{ SCNR}_{\text{peak}}\text{ and CA}_{t}\leq\text{ CA}_{\text{peak}}\\ \end{array} \right. \tag{13}\]  
After the agent starts to perform the task, its performance in the current environment is calculated once after each episode and rewarded or punished according to the preset reward mechanism, in Equation (13), \(\text{SCNR}_{\text{peak}}\) is denoted as the peak value of the signal-to-noise ratio in the total episode, \(\text{CA}_{\text{peak}}\) is denoted as the peak value of the clutter attenuation of the individual range cell in the total episode.