Learning Based Deadline Aware Congestion Control
Rongji Liao

Jinyao Yan

Communication University of China
Beijing, China
liaorongji@cuc.edu.cn

Communication University of China
Beijing, China
jyan@cuc.edu.cn

Tao LinYuhao Chen

Communication University of China
Beijing, China
lintao@cuc.edu.cn

Communication University of China
Beijing, China
chenyuhao@cuc.edu.cn

ABSTRACTA large variety of applications are deadline aware such as 360
degree VR (Virtual Reality), video conference and online gaming.
In this paper, we propose a congestion control algorithm using
deep reinforcement learning named DRL-CC to make the dataarrive before the deadline. The initial experiments validate that
our method dramatically increases the number of packets arrived
before the deadline compared with existing algorithms.

CCS CONCEPTS• Networks → Transport protocols.

KEYWORDS
Reinforcement Learning, Congestion Control, Deadline Aware
ACM Reference Format:
Rongji Liao, Jinyao Yan, Tao Lin, and Yuhao Chen. 2021. Learning Based
Deadline Aware Congestion Control. In 5th Asia-Pacific Workshop on Networking (APNet 2021) (APNet 2021), June 24–25, 2021, Shenzhen, China, China.
ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/3469393.3469675

1

PCC-RL [3] only consider the best effort congestion control basedon current network environment. For example, the reward functionof PCC-RL is related with packet loss, throughput and round-timetrip (RTT), with the goal of expecting the neural network to learncongestion control policy with low latency, high bandwidth and low
packet loss. It is foreseeable that PCC-RL may perform well when
dealing with ordinary congestion control scenario. However, when
there is a delivery deadline for each package, the goal of congestion
control is not only to do well in the terms of traditional congestion
control targets, but also to ensure that each package arrives before
the deadline to improve the performance of such applications.
This paper proposes a congestion control algorithm using deep
reinforcement learning to solve the problem of data transmission
with deadlines in this scenario. We obtain the information about
deadline for packets from application layer. Then DRL-CC makes
use of it to learn a policy of congestion control to increase thenumber of packets arrived before the deadline.

2SYSTEM DESIGNINTRODUCTION

Recently a large amount of applications such as 360 degree video,video conference and online gaming are emerging. These applications have deadline requirements for data delivery, that is, multimedia data must arrive at the receiving end before a given deadline.
If data is transmitted to the receiver out the scope of the deadline,
it will be useless. Previous study on congestion control do not consider the constraint of deadline, thus they perform not well in thisscenario.
Traditional congestion control algorithms such as Reno [2] andBBR [1] and machine learning based congestion control algorithms
This work was supported in part by the National Key Research and Development
Program of China(Grant No. 2019YFB1804300) and the National Natural Science Foundation of China (Grant No. 61971382).
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
APNet 2021, June 24–25, 2021, Shenzhen, China, China© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8587-9/21/06. . . $15.00
https://doi.org/10.1145/3469393.3469675

Figure 1: The framework of DRL-CC
As shown in Figure 1, DRL-CC consists of three components,
including application, network, and Deep Q-Network (DQN) algorithm, where we utilize deep neural network to approximate theQ-function Q(state,action) and calculate the optimal value for eachpossible action for given state. First, we retrieve the information
about deadline of packets from the application layer and the information about congestion obtained from the network environment

APNet 2021, June 24–25, 2021, Shenzhen, China, China

into the Packet Features module. Then the Packet Features module
will output the state and reward of the DQN, and store the data in astorage queue. DQN will decide the amount of change in the sending rate with given states. Finally, the Last State, Action, Reward,and State data are stored in the memory dataset named Experience
Replay in the DQN network for subsequent learning and updatingof the neural network.
Next, we define the state and action space of DRL-CC. Compared with PCC-RL, the state of DRL-CC is not only composedof congestion information such as packet loss, throughput andround-time-trip (RTT), but also includes packet deadline information where the packet id, deadline, and creation time of each packet
will be obtained. We consider the following features of a connection:(i) instant loss rate: packet number of loss in RTT/ packet numberof sent in RTT, (ii) loss rate: packet number of loss/packet numberof sent, (iii) instant acknowledgement rate: packet number of ack in
RTT/RTT, (iv) acknowledgement rate: packet number of ack/total
connection time, and (v) remain time of the packet, which is calculated by deadline, creation time, current time from application
layer. The action space consists of five actions on adjusting sending
rate, which is thus defined as:

Rongji Liao, et al.

Figure 2: Network topology

capacity is set to 15 Mbps for the bottleneck link. We use burst
background traffic generated from a real-world network data tosimulate the real-world network environment. Random packet loss
may occur on the link. The random packet loss rate is 0, 1%, and0.1%, where 1%, and 0.1% is set to simulate a wireless network.
Random packet loss is uniformly distributed. Our experimental
results are shown in Figure 3.

action = {1 + α, 1 − α, 1 + β, 1 − β, 1}We set α, β to 0.1, 0.2 in our experiment. The action of DRL-CC
is a number to get a new sending rate:sendinд rate = last sendinд rate ∗ action
The reward function of PCC-RL mainly considers the network
congestion information such as throughput, packet loss and RTT.
Therefore, the reward guides the neural network to achieve a good
performance congestion control without guaranteeing as many
packets as possible to arrive before the deadline. The reward function design of our algorithm not only considers the current network
congestion, but also considers the demand for packets arriving before the deadline. Therefore, the entire reward function consists of
two parts: the reward about network congestion which is calculatedby the condition of network congestion and the reward about thedeadline which is calculated by the condition of packet arrived
before the deadline.
total reward =rewardconдest ion cont r ol +rewardpacket ar r ived be f or e t he deadline
We expect our algorithm can achieve a good balance between
congestion control and deadline of packets.

3

PRELIMINARY RESULTS

Our experiment is based on the simulation system introduced inDTP [4]. The information of deadline is transmitted from application layer to the transport layer in DTP. Then, our proposedcongestion control algorithm makes use of the deadline information of the packet.
First, we design a network topology, as shown in Figure 2. There
are three senders in the simulation network. Two of them are usedas sender with a congestion control algorithm for comparison, while
the other is used to simulate background traffic. The three senders
share a link. Corresponding with three senders, receivers receive
data from senders. The propagation delay is set to 0.02s and theFigure 3: Performance Comparison
Figure 3 illustrates simulation results of four congestion algorithms including Reno, BBR, PCC-RL and our algorithm DRL-CC.
Figure 3 shows the ratio of the packets that arrived before thedeadline to the packets that have been sent. Reno, a congestion control algorithm based on packet loss, performs worse in congestion
control when the packet loss rate becomes bigger. Compared withother algorithms, our algorithm based on reinforcement learning
performs well at each packet loss rate, and the ratio that packets
arriving before the deadline to the number of packets sent has increased by about 10% 15% compared to BBR and PCC-RL. We cansee that our proposed deadline aware congestion control algorithm
achieves a significant improvement of performance, compared withthe baseline methods.

4SUMMARY AND FUTURE WORK

We propose DRL-CC to solve the problem of data transmission withdeadlines. The preliminary results validate that our method has
a potential to achieve a significant improvement of performancein terms of deadline. Our future work mainly focuses on packet
scheduling in the application layer and work with our current work
jointly to solve the problem of data scheduling and transmission
deadline.

Learning Based Deadline Aware Congestion Control

REFERENCES[1] Neal Cardwell, Yuchung Cheng, C Stephen Gunn, Soheil Hassas Yeganeh, and Van
Jacobson. 2017. BBR: congestion-based congestion control. Commun. ACM 60, 2(2017), 58–66.
[2] V. Jacobson. 1990. Modified TCP congestion avoidance algorithm. (1990).

APNet 2021, June 24–25, 2021, Shenzhen, China, China[3] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar.
2019. A deep reinforcement learning perspective on internet congestion control.
In International Conference on Machine Learning. PMLR, 3050–3059.
[4] Hang Shi, Yong Cui, Feng Qian, and Yuming Hu. 2019. DTP: Deadline-awaretransport protocol. In Proceedings of the 3rd Asia-Pacific Workshop on Networking
2019. 1–7.