# Improved DQN for Dynamic Obstacle Avoidance and Ship Path Planning
## 3 The Proposed Deep Q Network(DQN)
### The Improved DQN  
DQN uses prioritized experience replay (PER). When sampling data train the Q network, the data will be uniformly sampled from the playback buffer. This is not necessarily the best method because some data may be more critical. Assuming that some data have been sampled before, it was found that the timing difference error of these data is enormous. The timing difference error is the difference between the output of the network and the target, which means that these data are relatively challenging to train when training the network. Since it is not easy to train, we should give them a high probability of being sampled, that is, give them a priority. In this way, we will consider the data that are not good for training. When using PER, we will change the sampling process and the method of updating parameters because of changing the sampling process. So PER not only changes the distribution of sampling data but also changes the training process. Ordinary experience playback obtains a sample every time and uses it to update DQN parameters. Prioritized experience replay assigns a weight to each sample and performs non-uniform random sampling according to the weight. If DQN has an inaccurate judgment on the value of (\(s_{j}\), \(a_{j}\)), that is, \(Q\) is far from \(Q_{*}\), the sample should have a higher weight. The data generated during navigation have different levels of importance. In path planning, agents will generate many such samples. However, the ship will avoid obstacles during the navigation process, and these small samples will significantly impact the path. Therefore, the data on avoiding obstacles should have a higher weight and receive more attention. Samples should have different degrees of attention. We can determine which samples are more important according to the absolute value of TD error. If the absolute value of TD error \(\delta_{j}\) is large, which indicates that the current evaluation of the real value of \((s_{j},a_{j})\) is inaccurate. Then the higher weight should be set for \((s_{j},a_{j},r_{j},s_{j+1})\). In this paper, Equation (9) is used to calculate the sampling probability. If uniform sampling is used, all samples have the same learning rate of \(alpha\). If non-uniform sampling is adopted, the learning rate \(alpha\) must be adjusted according to the sampling probability. Set the learning rate as Equation (10). \[p_{j}\propto|\delta_{j}|+\epsilon \tag{9}\]  
\[p_{j}=\frac{p_{j}^{a}}{\sum_{k}p_{k}^{a}} \tag{10}\]  
where, \(\epsilon\) is a small number to prevent the sampling probability from approaching 0. \(\beta\) is a super parameter that needs to be adjusted. We take multi-step updating to make the Q network as close as possible to the target value. The multi-step method combines the Monte Carlo method and the time series difference method, so it has not only the advantages and disadvantages of the Monte Carlo method but also the advantages and disadvantages of the time series difference method. Previously, only a particular step was sampled, so the data obtained is accurate, and the Q value is estimated. Now there are more steps to sample and only N steps to estimate the value, so the impact of the estimated part will be relatively small. Of course, the disadvantages of the multi-step method are the same as those of the Monte Carlo method. Because there are many items, the total variance of N items will be more significant. However, we can adjust the value of N to balance the variance and the inaccurate Q value. The whole algorithm flow is shown in Algorithm 1.