# An Intelligent TCP Congestion Control Method Based on Deep Q Network
## 2 Related Works
### Reinforcement Learning and Its Applications  
Reinforcement learning (RL) is a subarea of machine learning. The basic idea of RL is to construct an agent to interact with the environment and learn the best policy to achieve the goal by maximizing the cumulative reward that the agent receives from the environment. In essence, network congestion control can be regarded as an optimization problem [18; 19; 20]. Reinforcement learning has been widely used in many important areas. [PERSON] et al. [21] proposed a novel energy management approach for real-time scheduling of an microgrid. [PERSON] et al. [22] proposed a reinforcement learning method to optimize the XSS detection model to defend against adversarial attacks. In the past, researchers have proposed a large number of optimization methods, such as NSGA-III variants [23]. Compared with traditional congestion control algorithms, the congestion control algorithm based on reinforcement learning has better adaptability and can learn new congestion control policies from the network environment independently. Due to this feature, reinforcement learning has become an important tool for network and protocol designers in recent years. The study of congestion control based on reinforcement learning has become a hot spot and has been applied in various areas. For example, [PERSON] et al. [24] suggests learning channel allocation decisions by using a linear bandit model to minimize total switching costs in multichannel wireless networks. [PERSON] et al. [25] uses deep reinforcement learning (DRL) to deal with large and complex state spaces when solving cloud resource allocation and power management problems. [PERSON] et al. [26] presents a DRL-based framework for energy-efficient resource allocation in cloud RAN. [PERSON] et al. [27] propose a reinforcement learning-based decision method for electricity pricing plan selection by smart grid end users. [PERSON] et al. [28] present a deep reinforcement learning method for energy-conscious optimization of edge computing. [PERSON] et al. [29] propose a deep inverse reinforcement learning method for autonomous driving in an urban environment. [PERSON] et al. [30] present a reinforcement learning (rl) control strategy for the participation in an incentive-based demand response program of a cluster of commercial buildings. [PERSON] et al. [31] use a reinforcement learning method to address the question of whether they can predict reaching movements in a full skeletal model of the human upper extremity. In addition, there are many algorithms based on reinforcement learning to improve the quality of service of network applications. [PERSON] et al. [32] presented an algorithm based on RL to generate congestion control rules to optimize QoE for multimedia applications. [PERSON] al. [33] defines the network resource allocation problem as a DEC-POMDP model in a multiuser video stream, and applies a distributed RL algorithm to solve the problem. [PERSON] et al. [34] uses RL algorithm to improve the QoE of video streams by adaptively changing parameter configurations. Its limitation is the use of a table-based algorithm, thereby limiting their application to large continuous domains. [PERSON] et al. [35] presented Hd-TCP: a custom congestion control algorithm that uses deep reinforcement learning to deal with the poor network experience caused by frequent network switching on high-speed networks from a transport layer perspective. All above scenarios are task driven. They are designed for specific applications and cannot be applied directly to congestion control problems. QTCP [10] is the first solution to directly apply RL to TCP congestion control protocol design. QTCP continuously updates the values of possible state-action pairs of protocols based on measurements of performance indicators collected from the network environment and uses the Q-Learning algorithm to search for the best action, that is, how to adjust CWND. In a particular state, maximize the long-term return of the sender.