# Reinforcement Learning-Based Anti-Jamming in Networked UAV Radar Systems
## 4 Algorithm Design
### DGQL (Double-Greedy Reinforcement Learning) Algorithm Based on Information Representation  
This section will further improve the Q-Learning algorithm based on the information representation theory and the greedy algorithm. The DGQL framework under the condition of networked UAV radar systems is given. Q-Learning is a commonly used reinforcement learning method for anti-jamming. The policy selection for Q-Learning is based on the \(Q\) table. The \(Q\) table has two dimensions: state and action. We consider all parameters of jammer 1, jammer 2 and target as the environment. The algorithm is required to give the corresponding policy of a radar system when given any environment that changes with time, so as to maximize the target information received by the UAV. Specifically, we consider the combination of jamming parameters at each time as the state, and the combination of radar system policies at each time is considered as an action. This process can be modelled as a Markov decision process (MDP). Traditional Q-learning uses tables to store \(Q\) values corresponding to each state and action, hence it has two limitations, namely scalability and continuous action. Therefore, deep Q network (DQN) and DDPG (deep deterministic policy gradient) methods were proposed in the reinforcement learning family. DQN improves the scalability of Q-learning by adopting neural network may generate \(Q\) value to avoid tabular representations. However, the learning effect of DQN is related to the memory stored previously in a database. Simulation experiments usually show that DQN has a better learning effect and convergence speed when the correct actions and wrong actions account for half of each in the memory, which is not the case in the model of this paper. DDPG is a reinforcement learning method for continuous action by integrating a deep convolutional neural network as the simulation of policy function and \(Q\) function, that is, policy network and \(Q\) network, and then use the method of deep learning to train the above neural network. However, fine tuning of parameters in DDPG is important, such as the learning rate of the actor network and critical network, regularization parameters, neural network structure and so on. If the parameters are not adjusted properly, the algorithm convergence behavior is poor. Therefore, as an initial contribution to adopt reinforcement learning in UAV-networked radar systems, we select Q-learning algorithm of ensured stability. This helps to gain more insights in the system model, problem dimension reduction and performance analysis, rather than devoting to fine-tuning the \"black-box\" in DQN and DDPG for high state-action parameter dimensions. Firstly, the algorithm calculates the SIR received by the communication UAV. This is based on the combination of various radar policies and jamming policies according to the network topology. According to the radar working mode, the algorithm judges the calculation method of information, and then stores the \(R\) value in reserve. Then the modified Q-Learning algorithm is used to iterate the policy. Considering that the moving position of the communication UAV will cause a large energy loss, if its action is not constrained, it will generate results that are not realistic. Therefore, we further constrain the action selection by the double greedy algorithm. The greedy algorithm is embedded in the Q-Learning algorithms to allow randomized action selection. It enables the agents to take randomized actions determined by probability distribution rather than \(Q\) table. This probability may be controlled by the threshold \(\varepsilon\) of the greedy algorithm. In our system, most of the energy of the communication UAV is used for motion, which should be controlled. Therefore, a soft constraint is set on the communication UAV that allows 1 or 2 moving steps with a higher probability and only set a low probability for further movement. We set two thresholds \(\xi_{1},\xi_{2}\), where \(0<\xi_{1}<\xi_{2}<1\).