# A Modified Quad Q Network Algorithm for Predicting Resource Management
## 2 Literature Review: Related Work
### Introduction to the Algorithms of Deep Q Learning  
Deep learning network consists of numerous float values, weights, and biases [33]. These values determine the deep learning output, and the initialization process is required because these values are not defined before the learning begins. Although the most commonly used method is to initialize with random values, Glorot Initialization allows one to create an initial weight value with more stable results. This method is also called Xavier Initialization because it was proposed by [PERSON]. In Deep Q Network (DQN), the idea of target network is applied. If one looks again at the expression that updates the Q value in Q-Learning, one can see formula \(Q(s,a)=Q(s,a)+\alpha(R+\gamma maxQ(s^{\prime},a^{\prime})-Q(s,a))\). The DQN has an idea called target network [34]. In the formula that updates Q value in Q-Learning, the \(\alpha\) represents the learning rate. In the update formula, Q(s,a) is modified by multiplying \(R+\gamma maxQ(s^{\prime},a^{\prime})-Q(s,a)\) by \(\alpha\). If \(\alpha\) is 0, then \(Q(s,a)\) will not change, and if the \(\alpha\) is 1, then \(Q(s,a)\) will be \(R+\gamma maxQ(s^{\prime},a^{\prime})\). Because \(\alpha\) is usually a value between 0.0 and 1.0, \(Q(s,a)\) tends to \(R+\gamma(maxQ(s^{\prime},a))\) as the learning continues. Here, the problem of using network weights with the same goal as the current value arises if weights and biases of the Q Network are expressed as \(\theta\). The weight of the Q Network changes each time one learns. Goals change as well, thus, one might have difficulty in converging to a certain value. Therefore, we need to create a separate network with the same structure as the Q Network called target network, and mark it with a symbol \(\theta^{-}\), as in the formula \(Q(s,a;\theta)\to R+\gamma maxQ(s^{\prime},a^{\prime};\theta^{-})\). The target network retains a fixed value during the original Q Network learning for a few times and periodically resets it to the original Q Network value [35]. This can be an effective way of learning because Q Network can be closer to a fixed target network. DeepMind's [PERSON] cited three major improvements after on the DQN, which he has presented to Nature in his fourth talk, \"Deep Reinforcement Learning\". They are Double DQN, Prioritized replay, and Dueling DQN. This paper introduces about Double DQN and Dueling DQN with direct calculation change. Double DQN was originally an algorithm using two Q Networks to improve the problem of Q-Learning in 2010, before DQN came out. Double Q-Learning shows that Q-learning is sometimes poor because Q values tend to be overvalued. If one looks closely at the \(max\)\(Q(s^{\prime},a^{\prime})\) in the original Q-Learning expression, one can see that this value, given the status \(s^{\prime}\), selects \(a^{\prime}\), which has the highest Q value in the Q Network, and multiplies the Q value by gamma to create a target value for \(Q(s,a)\). However, the problem is that one has to use the same Q Network when selecting and importing Q values [36]. The selected Q value is usually large value. Once it is imported, the Q value goes in an increasing direction. However, if the Q value becomes larger than necessary, the performance of the Q Network will be greatly reduced. To prevent this, the first method proposed by [PERSON] is to use two Q Networks [37]. The key to Double DQN is that these two Q Networks can help prevent unnecessary growth of Q values. As we can see above, DQN already had the idea of target network. The DQN of formula \(Q(s,a;\theta)\to R+\gamma maxQ(s^{\prime},a^{\prime};\theta^{-})\) is replaced by the Double DQN of formula \(Q(s,a;\theta)\to R+\gamma maxQ(s^{\prime},argmaxQ(s^{\prime},a^{\prime}; \theta);\theta^{-})\). Once one action is selected, using the existing Q network target network, Q value is obtained. Dueling DQN is the result of the network before one prompts for A value Q. A and V are then divided by the idea back together. The important thing is that one can calculate the V here.