# Study on Reinforcement Learning-Based Missile Guidance Law
## 4 Reinforcement Learning  
Prior research on RL-based missile guidance was studied under a very limited environment and did not show whether a single network can cover a wide environment and initial conditions which push the seeker to the hardware-limit or not. Additionally, the comparison had been implemented under an unfair condition for the comparison target, which each of them does not take identical states. In this paper, it is demonstrated that the RL-based guidance is able to cover a wide environment under plausible wide initial conditions, and with various system models. In this section, a novel structure and methodology for training will be proposed. We adopted a Deep Deterministic Policy Gradient (DDPG)-based algorithm [8]. DDPG is a model-free reinforcement learning algorithm that generates deterministic action in continuous space, which makes off-policy training available. The NN we used was optimized via gradient descent optimization. Since the environment we are dealing with is vast and can be described as a sparse reward environment, conventional binary rewards nor even a simple shaped reward do not really help to accelerate training. Thus, the algorithm we used in this paper was slightly modified by making every step of an episode get an equatio  
Figure 4: Conceptual diagram of proportional navigation guidance. Figure 5: Missile simulation schematic diagram with proportional navigation guidance. as a reward. It helps the agent to effectively consider the reward of the far future. Figure 6 is an overview of the structure we designed. The agent feeds \(\Omega\) and \(V_{m}\) as input and stores a transition dataset, which consists of state, action, reward, and next state into a replay buffer. Developed policy gradient algorithm using an artificial neural network (ANN) needs two separated ANNs which are the actor that estimates the action and the critic that estimates the value. Figure 7 shows each structure. DDPG, generally, calculates its critic loss from the output of the critic network and the reward of a randomly sampled minibatch, and updates the critic and the actor every single step. Q is the action value, and is the hyperbolic tangent activation function. Both consist of three steps of hidden layers, and actor-network has \(\pm 10\) g of clipping step as we set. We used the Huber function [13] as a loss function that shows high robustness to outlier and excellent optimization performance. The equation is as follows:  
\[L_{\delta}(y,f(x))=\frac{1}{n}\sum_{i=1}^{n}\left\{\left\{ \begin{array}{ll} \frac{1}{2}(y_{i}-f(x_{i}))^{2},&if\left|y_{i}-f(x_{i})\right|\leq\delta\\ \delta\Big{(}\big{|}y_{i}-f(x_{i})\big{|}-\frac{1}{2}\delta\Big{)},&otherwise \end{array} \right\}\right. \tag{11}\]  
The training proceeds with a designed missile environment. The environment terminates the episode if it encounters one or more termination conditions as follows--Out of Range condition: target information unavailable since the target is outside of the available field of view of the missile seeker; Time Out condition: time exceeds the maximum timestep which is 400 s; there is also a Hit condition (\(if\ R<5\)): judged that the missile hits the target. However, the last condition is not for termination, but exists as an observable data that decides whether the policy is proper for the mission or not. As an episode ends, each of the transition datasets gets the corresponding reward which is  
Figure 6: Structure of the training algorithm. Figure 7: Actor-network and critic-network. An aspect of the case that satisfies the Time Out condition while Out of Range condition is not satisfied is the maneuver in which the missile is approaching infinitely in spiral trajectory, which made us aware that the magnitude of the cost was becoming too big. Thus, we considered the maneuvers as outliers and gave them the lowest score and stored them to the replay buffer.