## 2 Background Principle
### Principle of Deep Q-Network  
Q-learning, the reinforcement learning underlying a DQN, teaches agents that interact with the environment. In the early stages of learning, the agent learns slowly over time, taking random actions in various situations called'states'. Policies are configured by updating all values to the table-type Q-table, depending on the status operation, and then referring to them and selecting the action. This is expressed in Figure 3 and Equation (3) as a way to update the Q-table to maximize the final cumulative reward while sequentially progressing the episode [13]. \[Q(s,a)\ =r(s,a)\ +\gamma\ \max_{a}Q(s^{\prime},a^{\prime}) \tag{3}\]  
The sum \(Q(s,a)\) of all rewards that can be received when taking action a in the current state \(s\) can be calculated as the sum of the immediate rewards that can be taken and received in the future. On the right side of the equation, \(r(s,a)\) represents the immediate reward value to be received when action a is taken in the current state s. In addition, \(s^{\prime}\) is the very next state that is reached by taking action \(a\) in the current state \(s\), and \(\max_{a}Q(s^{\prime},a^{\prime})\) is the maximum value of the reward that can be received in the next state \(s^{\prime}\). The goal of the agent is to select an action that maximizes this value. In addition, \(\gamma\) is a value called a discount, which controls the importance of a future value. The larger the value of \(\gamma\) is, the greater the value of the future reward, and the smaller the value is, the greater the importance of the immediate reward. Because Q-learning must have all values for a state-action corresponding to the Q-table, there is a limit to how many states can be stored. To improve this, there was a way to learn Q-learning using neural network models; however, when applying high correlation  
Figure 3: Q-learning process. Figure 2: Decoding process of erasure coding. problems between learning data and gradient descent methods, the target value, Target, is moving. Therefore, using neural network models instead of a Q-table, DQN combines Q-learning with deep learning, enabling neural network models to approximate the Q values, and constructs the target neural networks separate from a replay of the experience. This is expressed in Figure 4 and Equation (4) [14]. \[Cost=\Big{[}\Big{(}r(s,a)\ +\ \gamma\ \underset{a}{\max}Q(s^{\prime},a^{\prime}; \ \theta^{\prime})\ -\ Q(s,a;\ \theta)\Big{)}\Big{]}^{2} \tag{4}\]  
In the existing Q-learning in Equation (3), the final goal is to converge and reach the true value when the left and right sides are the same. The DQN shown in Equation (4) also has the same goal, i.e., to learn the neural network models in the direction of minimizing the difference between the two values through the loss function of the neural network models. In addition, in Equation (4), \(Q(s^{\prime},a^{\prime};\ \theta^{\prime})\) is the Q-Target and refers to the state, action, and parameters used in a neural network applied in deep-learning-based reinforcement learning. The target network optimizes \(\theta\) of the real network, holding the weight of the value of the Q-Target \(\theta^{\prime}\), even when minimizing the loss function, similar to how it holds the target when conducting supervised learning. In other words, when learning a predictive network representing \(Q(s,a;\ \theta)\), as shown in Figure 4, a target network representing \(Q(s^{\prime},a^{\prime};\ \theta^{\prime})\) is created separately to fix the weights. It then passes the weights of the forecasting network to the target network at every step. This learning is not affected by the weights the target is learning when narrowing the difference between the values of the target network and the predictive network. Therefore, more stable learning is possible than with Q-learning.