# An Intelligent TCP Congestion Control Method Based on Deep Q Network
## 3 Background  
### Dqn  
Q-learning [11] is a value-based reinforcement learning algorithm, which is the basis of DQN. Generally speaking, reinforcement learning interacts and learns from the environment by using an agent. The agent tries to generate a suitable action to maximize the received reward in a particular state, and finally learns the optimal control policy. The core idea of reinforcement learning is trial and error mechanism and policy optimization. Agents improve their policy by trying various actions to learn the good action that can get more reward from the environment. In Q-learning, a greedy policy is used to generate the action \(a\), which means the agent will always choose the action that can obtain the maximum Q value in each state \(s\). The core idea of Q-learning is the Q(\(s\), \(a\)) function. The Q(\(s\), \(a\)) function represents the cumulated reward until the current episode ends after choosing an action \(a\) in state \(s\). According to the Bellman equation, the Q value can be iteratively calculated by Equation (1). \[\text{Q}(s_{t},a_{t})\leftarrow\text{Q}(s_{t},a_{t})+\alpha(\text{R}(s_{t},a_{ t})+\gamma\text{max}_{a}\text{Q}(s_{t+1},a)-\text{Q}(s_{t},a_{t})) \tag{1}\]  
where \(\alpha\) is the learning rate; R is the reward value obtained after performing the action \(a_{t}\); \(\gamma\) represents the discounting factor, which determines the importance of future rewards. In theory, all Q(\(s\), \(a\)) values can be held in a two-dimensional table, and all the exact Q values can be calculated by iterating enough times according to Equation (1). After calculating all Q values in the Q table, the optimal control policy is trivial, which can be implemented by choosing the action \(a\) which can maximize the Q value at each state. However, in practical, the Q table is too large to hold in memory. Therefore, a function is used to approximate the Q table, which means the Q value can be estimated if a \((s,a)\) pair is inputted to the function. The simplest way is using a linear function to approximate the Q table. To get a better result, a non-leaner function such as artificial neural network can be used to approximate Q table. However, with function approximation, the oscillation problem may be occurred during the training process. As shown in Figure 1, DQN [12] is the development of Q-learning algorithm [13], which is a new algorithm combining deep learning with reinforcement learning. DQN solves the oscillation problem during training by experience replay and dual neural network. Different from Q-learning, the parameters are updated immediately after gaining experience in each step, DQN saves the experience \(e_{t}=(s_{t},a_{t},r_{t},s_{t+1})\) obtained in each step to an experience pool, which is formalized as \(\text{D}_{t}=\{e_{1},\dots,e_{t}\}\). DQN will randomly sample experience from the experience pool to update the parameters. This mechanism has three advantages:  
1. Each experience may be used for multiple parameter updating. Combing with the experience replay, the learning is more effective. 2. The samples of Q-learning learning are generated in continuous actions, which makes the samples have strong correlation with each other and result in oscillation in training. DQN randomly samples from the experience pool to eliminate the correlation between continuous samples and enhance stability of the training process. 3. DQN randomly picks samples from the experience pool for training, which avoids the problem that Q-learning may fall into a local optimal solution. In addition, DQN constructs two neural networks with the same structure for learning. They are: Q-network and target-Q network. Samples are generated by the target-Q network, and the Q-network is used for parameter update. After c-time parameter update, the parameters of Q-network are copied into target-Q network. This leads to more stable training because it keeps the target function fixed (for a while).