HierCC: Hierarchical RDMA Congestion Control
Jiao Zhang1,3 , Yali Zhang2 , Zixuan Guan2 , Zirui Wan1 , Yinben Xia2 , Tian Pan1 ,Tao Huang1,3 , Dezhi Tang2 , Yun Lin2
BUPT1 , Huawei2 , Purple Mountain Laboratories3

ABSTRACT

1

RDMA has been increasingly deployed in data centers todecrease latency and CPU utilization. However, existing
RDMA congestion control schemes fail to address instantaneous large queue build-up or bandwidth under-utilization
associated with frequent traffic bursty. In this paper, we
argue that traffic uncertainty is the essential reason that
constrains data center congestion control from simultaneously achieving high throughput and deterministic latency.
Since aggregated flows within the same rack are relatively
long-lived, we propose HierCC, which aggregates flows destined to the same IP in a rack and hierarchically controls therate of flows. The rate of aggregate flows between racks iscontrolled by a credit-based congestion control mechanism.
Then the bandwidth obtained by an aggregate flow in a rackis allocated to the corresponding individual flows from thatrack promptly and accurately. We evaluate HierCC using SystemC and large-scale NS3 simulations. Results indicate that
HierCC can significantly mitigate buffer usage and reducethe 99th percentile FCT by up to 20% and 40% compared with
HPCC and DCQCN under a realistic workload, respectively.

With the proliferation of distributed storage and machinelearning applications and the trend of resource disaggregation in data centers, data center transmission control is
continuously seeking low or even deterministic end-to-end
latency. In addition, data center link speed has grown from
1Gbps to 100Gbps these years, and high-speed packet processing takes up a higher CPU overhead. Therefore, RDMA
is increasingly deployed to reduce the CPU overhead causedby the end host software stack and the processing latency
caused by multiple memory copies [10]. Meanwhile, RDMA
congestion control has received much attention to further
reduce the in-network latency caused by traffic congestion.
Congestion control mechanisms have been studied formany years, including sender-driven [16, 17, 29] , receiverdriven [7, 11, 18], and switch-driven solutions [8, 14, 24].
The existing mechanisms undoubtedly have good performance in the scenarios where all flows are long-lived flows,
however, they still fail to address the instantaneous queuebuild-up or bandwidth under-utilization associated with frequent arrival and completion of short flows. Most existing
RDMA congestion control mechanisms use the line-rate as
the initial transmission rate. The arrival of multiple shortflows easily cause a large queue build-up. And after theseshort flows finish, the remaining flows will take multiple
rounds to occupy the bandwidth released by the completed
flows. Due to this frequent convergence process, it is difficult
to guarantee high throughput as well as near-zero queuinglatency simultaneously.
The dilemma of high throughput, as well as ultra-low latency, has been addressed to some extent by existing congestion control from a variety of perspectives. However, due tothe unpredictability of traffic and high flow churn in data centers, it is difficult to avoid both transient queue accumulation
and bandwidth under-utilization. For example, if we keep
near-zero queue length, then bandwidth under-utilization
will happen when short flows finish since the rate of theremaining flows does not fill the full bandwidth and there
are no additional packets to send. In this paper, we argue
that the high traffic uncertainty in data centers is the key
challenge of designing a RDMA congestion control mechanism with both high throughput and determining latency.
But accurately predicting the arrival and finish of each flow

CCS CONCEPTS• Networks → Transport protocols; Data center networks.
ACM Reference Format:
Jiao Zhang1,3 , Yali Zhang2 , Zixuan Guan2 , Zirui Wan1 , Yinben
Xia2 , Tian Pan1 ,, Tao Huang1,3 , Dezhi Tang2 , Yun Lin2 . 2021. HierCC: Hierarchical RDMA Congestion Control. In 5th Asia-PacificWorkshop on Networking (APNet 2021) (APNet 2021), June 24–25,
2021, Shenzhen, China, China. ACM, New York, NY, USA, 8 pages.
https://doi.org/10.1145/3469393.3469396
Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than ACM must be honored. Abstracting withcredit is permitted. To copy otherwise, or republish, to post on servers or toredistribute to lists, requires prior specific permission and/or a fee. Request
permissions from permissions@acm.org.
APNet 2021, June 24–25, 2021, Shenzhen, China, China© 2021 Association for Computing Machinery.
ACM ISBN 978-1-4503-8587-9/21/06. . . $15.00
https://doi.org/10.1145/3469393.3469396

INTRODUCTION

APNet 2021, June 24–25, 2021, Shenzhen, China, China

is also extremely hard in data center networks. Fortunately,
it has been argued that aggregated flows within the samerack are relatively long-lived [21].
Based on this investigation, we propose HierCC, which
aggregates individual flows destined to the same IP in a
rack and controls the rate of aggregate flows between racks
using a credit-based proactive congestion control. Since theaggregate flows are generally long-lived, the unexpected
converging process can be significantly reduced. Then the
bandwidth obtained by an aggregate flow in a rack will beallocated to the corresponding individual flows in that rack.
In this way, more prompt and accurate rate control can bemade since the response loop becomes much shorter and theflow information can be more easily obtained within a rack.
HierCC enables intra-rack queuing with an upper boundthrough more timely and accurate congestion control withina rack, and for inter-racks, the maximum queuing at thereceiver-side ToR to each receiver is Number r ack ×C. WhereC stands for the initial rate of an aggregate flow. This is muchsmaller than existing congestion control methods with bursty
flows. Moreover, the queue accumulation at the receiverside ToR only occurs when some aggregate flows are newly
created.
We evaluate HierCC using SystemC and large-scale NS3
simulations. Results show that the queue length in HierCC
is smaller than DCQCN by up to about 10x with incast or
realistic traffic pattern and high bandwidth utilization isachieved at the same time. HierCC can also reduce the 99th
percentile Flow Completion Time (FCT) by up to 20% and 40%
respectively compared to HPCC and DCQCN in large-scale
simulations.
In summary, we make the following contributions:
1) We investigate that traffic uncertainty is the essential
reason that constrains data center congestion control fromsimultaneously achieving high throughput and deterministic
latency. We also point out that aggregating flows, reducing
control loop and using proactive transport are three effective
ways to improve performance.
2) As far as we know, HierCC is the first hierarchical
RDMA congestion control scheme. We aggregate flows toease traffic uncertainty and control aggregated flows between racks and individual flows within racks in differentways.
3) We implement HierCC on SystemC and verify that
HierCC can significantly mitigate buffer usage and reduce
the flow completion time under a realistic workload.

X.et al.

2 BACKGROUND AND MOTIVATION
2.1 Goal: Ultra-low Tail Latency
Ultra-low tail latency has been more critical for datacenter
networks as the development of cluster-wide storage systems like NVMe [13, 15], machine-learning systems [26],and resource disaggregation [9]. A lot of attempts has been
conducted to reduce the end-end latency to satisfy the requirements of these new trends. RDMA has been graduallyemployed to mitigate the latency caused by the software
network stack at end hosts. But reducing the end-end flow
latency still relies on RDMA congestion control mechanisms.
Especially, as a forthcoming new datacenter architecture
paradigm, disaggregated datacenters require a network fabric to effectively connect different kinds of resource types
like CPU, GPU, memory, and disk. It is stated that 3 − 5us
network latency and 40 − 100Gbps network bandwidth arerequired to maintain application-level performance, and the
network congestion should achieve zero or close-to-zero
queuing [9]. That is to say, we require an elaborate RDMA
congestion control mechanism to guarantee quite small andstable queuing latency as well as high throughput even in
the circumstances with bursty traffic.

2.2

Traffic Uncertainty

Most flows in data centers have a short length and last for asmall period of time. It is reported that in Facebook’s Hadoop
cluster the length of about 60% flows is smaller than 1KB
and about 50% flows last less than 0.1 second. In the web
server cluster, about 80% flows are shorter than 10KB and
53% flows last less than 1 second [21]. Thus, the flows indata centers are quickly fluctuant. The uncertainty of thedata center traffic poses great challenges to the congestion
control.
Existing RDMA congestion control mechanisms generally
use the line rate as the initial rate [16, 17, 29]. However, they
will lead to a large queue buildup with bursty traffic, which
will correspondingly cause high end-to-end latency jitter.
We conduct a NS3 simulation to demonstrate the impactof uncertainty traffic. In this simulation, we use a FatTree
topology with 16 Core switches, 20 Agg switches, 20 ToRs.
Each ToR switch connects 16 servers via 25Gbps links andthe links between switches are all 100Gbps. The delay oflinks are all 1 µs and thus the RTT is 12 µs. We run two longlived flows as the background traffic and add 16 short 20KB
flows at 100µs as concurrent burst flows. HPCC is used as thestate-of-the-art congestion control scheme. Figure 1 shows
the variation of the queue length and overall throughput.
Figure 1(a) shows that an Incast traffic with the line-rate
as the initial rate will cause a queue length peak up to 340KB
at the switch. However, after the disappearance of the instantaneous bursty traffic, the other flows require several RTTs

6us RTT
12us RTT

100 200 300 400 500 600

Time(us)(a) Qlen

25
20
15
10
5
00

6us RTT
12us RTT

100 200 300 400 500 600

Time(us)(b) Throughput

Figure 1: Traffic burst causes performance deterioration with HPCC.
to climb back to the fair rate. This arises as a consequenceof the additive window increases for maxStaдe rounds in
HPCC. Figure 1(b) shows that long-lived flows will spend
up to about 70 µs to grab bandwidth freed up by short flows,which may greatly impact long throughput-sensitive flows.
The above micro benchmark shows that existing congestion control algorithms can rapidly converge to the ideal
point when handling fixed traffic pattern, but perform poorly
when traffic pattern changes rapidly. However, the flows indata centers are quickly fluctuant. Thus, the root cause of
the congestion control algorithm’s inability to accurately andresponsively allocate network bandwidth in data centers liesin the frequently changing traffic pattern.

2.3

Basic Idea

In order to reduce the impact of traffic uncertainty, we advocate that our congestion control algorithm should havethree properties: 1) Aggregated flows. 2) Short control loop.
3) Proactive rate allocation.
Aggregated flows. Traffic bursts are caused by quite shortand concurrent 5-tuple flows in data center networks. Congestion control mechanisms are hard to control the shortflows that possibly last for only one RTT. However, the destination servers in some data center services are relatively stable. For example, in the distributed storage service, multiple
senders write data to the disk array servers which make connections from some senders to the fixed disk array servers.
Therefore, if we aggregate all the connections from the samerack to the same destined disk array server, then the aggregated flows will be more long-lived. According to the
Facebook’s Cache follower cluster data [21], the aggregatedflows by destination host are tightly distributed around 1MB,
while the CDF of the Cache follower flow duration showsthat the length of half of the 5-tuple flows is less than 20KB.
Therefore, if we control the transmission rate in the granularity of an aggregated flow instead of a single flow, then thecontrol can be more effective and accurate.
Short control loop. Most existing mechanisms work on
an end-to-end basis, which is detrimental to making very

Queue length peak(MB)

350
300
250
200
150
100
50
00

APNet 2021, June 24–25, 2021, Shenzhen, China, China

Throughput(Gbps)

Queue Length(KB)

HierCC: Hierarchical RDMA Congestion Control

9
8
7
6
5
4
3
2
1
0

6us RTT
12us RTT

50

100

150

200

Number of flows

250

Figure 2: Congestion extent is positively related to the Incast scale with
HPCC.

prompt adjustment to sending rates. Severe network congestion or long network under-utilization will likely happen
as shown in the above analysis. We adjust the latency ofeach link from 1us to 0.5us and conduct the same simulation as above. Figure 1 shows that with the smaller RTT, thequeue length peak will also reduce from 340KB to 250KB
and the overall throughput gap will reduce from about 70µs
to approximately 35µs. It indicates that as the latency of the
feedback loop decreases, the sender will acquire link information more quickly and converge to the fair rate more rapidly.
Thus, the performance of flows can be improved.
Besides, we conduct simulations with a different numberof concurrent flows (16, 32, 64, 128, 256) in the Incast traffic.
Figure 2 shows that the queue length peak increases from
0.8MB up to 9MB with 12us RTT, and the queue length is
up to about 6MB with 6us RTT. This is because the congestion feedback all comes from the last hop. As the number ofconcurrent flows increases, the congestion extent at the last
hop also increases. Therefore, it is hard to guarantee a determined end-to-end latency since the queue accumulation ispositive to the number of concurrent flows.
Correspondingly, we conclude that shorter control loop
and limited control space are desirable for congestion control
schemes to achieve determined performance.
Proactive rate allocation. Passive congestion control mechanisms [16, 17, 29] adjust sending rates after congestion has
occurred and generally require multiple rounds to converge
to the ideal point, which is the primary cause of performance deterioration with varying traffic. In contrast, proactive congestion control algorithms [7, 11, 18, 19, 28] proactively avoid congestion by explicitly allocating the network
bottleneck bandwidth among passing flows, which greatly reduces wasted bandwidth when traffic patterns change rapidly.
In addition, proactive transports enable to limit buffer usage
by proactively restricting packets sending. Thus, we use theproactive rate allocation scheme to achieve fast convergence inour design.

APNet 2021, June 24–25, 2021, Shenzhen, China, China
Core1

X.et al.

Core2
C

ToR1
VOQ
Scheduler

VOQ
Scheduler

VOQ1.7

ToR4

C

ToR2

WRR
Scheduler

VOQ2.7

WRR
Scheduler

VOQ1.8

S1

S3

S2

S1

S7

S2

S7

S4

S1

S8

S8

S7

S3

S7

C

CreditServerFigure 3: Framework of HierCC.

3

DESIGNThe basic idea of HierCC is to aggregate flows at rack-level
while providing hierarchical control within the first-hop tierand between racks. HierCC greatly alleviates the first RTT
problem[12] by aggregating flows and using a pre-credit
mechanism. It can achieve stable ultra-low latency as wellas high throughput even under bursty traffic.
According to the analysis of aggregated flows in §2.3, weaggregate flows at the sender-side ToR and then propose a
rate allocation mechanism to control the rate of aggregate
flows between ToRs (§3.2). Then the sending rate of eachindividual flow is controlled based on the overall rate of theaggregated flow within each sender-side rack (§3.3). Since
the sender-side ToR is only one hop away from the sending
servers, the control loop becomes quite short and the control
space is limited. Even if the traffic is still uncertain within arack, it is possible to control the burst traffic more promptly.

3.1

1 and V OQ 2.7 at ToR 2. The packets coming from V OQ 1.7
and V OQ 2.7 will enter into the downstream output queue
to server 7. Each ToR switch has 1 VOQ scheduler and 2
downstream output schedulers. The downstream scheduler
determines how many credits will be sent to each corresponding VOQs according to the network status. Besides,
each VOQ has an initial amount of credits, denoted as C 0 . If
a VOQ has available credits, then it can dequeue the sameamount of packets. Since the traffic in each VOQ is relatively
stable, the credits assignment will be more effective without
suffering frequent rate converging process.

Framework

In HierCC, each ToR switch has Virtual Output Queues(VOQs) at the sender side and downstream output queues
to servers at the receiver side. Let M and N represent the
total number of ToR switches and servers. Each ToR switch
connects to k servers. All the packets coming from ToRi (i ∈ [1, M]) with destination end host j (j ∈ [1, N ]) willenter into V OQ i .j , that is, V OQ i .j has the aggregated traffic destined to host j. Thus, each ToR has at most N VOQs.
Besides, at the receiver-side, each ToR switch has k WRRschedulers to determine when and how many credits will besent to the corresponding VOQs. After receiving credits, thepackets in the VOQ will be transmitted.
We use a small two-layer leaf/spine data center topology to illustrate the main framework of HierCC in Figure
3. There are M = 4 ToR switches and 2 core switches. EachToR switch connects to k = 2 servers. Thus, the total number of servers is N = M × k = 8. Each ToR switch has 8
VOQs and 2 downstream output queues. For example, ToR
1 has V OQ 1.1 -V OQ 1.8 and ToR 2 has V OQ 2.1 -V OQ 2.8 . The
packets destined to server 7 will enter into V OQ 1.7 at ToR

3.2

ToRs mechanism

Packet transmission at sender-side ToRs. As describedabove, each ToR uses VOQs to accommodate the packets tothe same destination host. Each VOQ has an initial amountof credits C 0 in unit of bytes. In HierCC, a credit means that∆ = 1KB data can be sent at the sender-side ToR. When a
new credit packet to V OQ i j is received from the receiverside ToR, the amount of credits of V OQ i j will be increasedby the received credit amount. The number of flows in the
V OQ i .j , denoted as Fi j , is locally recorded. Besides, the total
bytes that have been sent from V OQ i j is also recorded as Si j .
The ToR VOQs scheduler employs the round robin method
to dequeue packets from different VOQs that has available
credits. Before dequeuing a packet from a VOQ, the valueof Fi j and Si j will be inserted into the packet header. Afterdequeuing a packet, the credit amount will be decreased by
the byte length of the packet.
WRR credit scheduler at receiver-side ToRs. HierCC
schedules the credits at the receiver side ToR switch. We
add a scheduler at each output port and this scheduler maintains a table of all the VOQs which send packets to this
output port. After receiving a packet, the receiver-side ToR
will determine which ToR the packet comes from. Then thenumber of sent bytes Si j and number of flows Fi j carried in
the packet header will be recorded in the table accordingly.
The receiver-side ToR also locally records how many credits
have been sent to V OQ i j , denoted as Cisentin the table.
j
To achieve flow-level fairness, the credits are sent to different V OQ i j based on the Weighted Round-Robin (WRR)
scheduler. The weight corresponding to each V OQ i j equalsFÍ i j . During the WRR scheduling, the output port calls one
i Fi j
VOQ’s information from the table each time. If Cisent< Si j ,jthen one credit will be sent to V OQ i j . On the contrary, if
Cisent≥ Si j , the output port has already sent enough creditsjfor this VOQ and the output port scheduler will directly jump
to the next VOQ sequentially, which means no more credits
will be allocated to V OQ i j until Cisent< Si j .
jHierCC: Hierarchical RDMA Congestion Control

3.3

APNet 2021, June 24–25, 2021, Shenzhen, China, China

Rate of the First Hop

Passive congestion control like DCQCN converges quite
slowly. Thus, we propose an explicit rate computation method
to control the first hop rate in HierCC. The first hop rate control is actually a one-hop many-to-one rate control problem.
The outgoing rate of the VOQ equals the incoming credit
rate. Thus, one straightforward method is to use the outgoingrate divided by the number of flows, that is, R(t) = Rc rNed(ti t) (t ) .
R(t) represents the rate of one flow at time t and N (t) is thenumber of active flows at time t.
In HierCC, we compute N (t) by recording the arrival ratee (t )
to the VOQ similar to RCP [8], that is, N (t) = R arR(tr iv−1).
Meanwhile, we need to control the queue build-up within asmall value. Thus, the rate of a flow is computed as R(t) =Qv oq (t )R c r ed i t (t )−β × RT T

hop
R(t − 1) ×
. Qvoq (t) stands for the queue
R ar r iv e (t )length of the VOQ. RTThop is the round trip time between a

Q

(t )

sender and the sender-side TOR switch. RTv oq
Thop means therate required to drain out the queue build-up in one RTThop .
Q(t )β × RTv oq
Thop (β ∈ (0, 1)) represents that the queue build-up

will be drained out in β1 RTThop . Thus, β is usually smaller
than 0.5 since using one RTThop likely causes instability.
We are not done here. Different from previous explicit rate
control mechanisms like RCP, Rcr edit (t) received by a VOQ
in HierCC is not a constant value. It is limited by both thesending rate of the VOQ and the receiver-side TOR allowed
rate. Thus, if the summation of the arrival rate is smaller thanit could, Rcr edit (t) will equal R ar r ival (t) and thus achieve awrong under-utilization equilibrium. To avoid this problem,

R(t) =


Rl ineR(t − 1) ×R cr ed i t (t )−β ×

Qv oq (t )−q 0
RT Thop

R ar r iv e (t )if Qvoq < 0otherwise(1)

Where q 0 is a constant value that the VOQ queue length will
converge to.
Then how to obtain Rcr edit (t) and R ar r ive (t) timely? The
sending rate is controlled in the granularity of one RTThop .
Since RTThop is quite small like 2us, counting how many
credit or data packets are received during a RTThop possibly
lead to 0 and a relatively large value intermittently. Thus, we
calculate Rcr edit and R ar r ival every β1 RTThop for accuracy.
At last, flows possibly burst into the network with the initial line-rate if multiple concurrent flows are established withthe same destination server. That is, the instantaneous queue
length in the sender-side ToRs possibly becomes quite high.
To avoid the large queue accumulation at the sender-side
ToRs, we stop the source from sending packets by setting
R(t) = 0 as long as the VOQ queue length is greater thanVhiдh . In this way, the momentary queue build-up can be

avoided. When the queue length is less than Vlow , The previous rate value before stopping the source will be sent to thesenders to resume transmitting packets.

4

IMPLEMENTATION

Sender-side implementation. A sender marks one reserved
bit in the data packet header every 10us. The sender-side
ToR then feeds a rate allocation packet for every marked
packet based on Equation (1). In this way, sender-side ToRs
donot need to maintain a flow table to store the informationof active senders. Besides, the sender NIC does not requirecomplicated algorithms, it only needs to adjust its sending
rate accordingly after receiving the rate allocation packet.
VOQs implementation. For a data center with N servers,
each ToR needs to maintain N VOQs. Currently, some switch
chips like Jericho [4] and Silicon One [2] can support 100K
VOQ queues [31]. Thus, HierCC works fine for data centerswith the number of physical servers below this specification,
but there are deployment issues in larger data centers. As
an exploration of determined latency solutions, HierCC isinitially designed with certain limitations to the specification,
which will be addressed in our subsequent work. Our initial
proposals include:
1) Divide large-scale data centers into multiple Pods, eachwith a few K servers. Deploy HierCC independently withineach Pod and use another control layer between Pods.
2) Design a VOQ aggregation scheme. When the traffic is
so heavy that it is about to exceed the VOQ specification, weaggregate the VOQ at the source ToR so that it goes fromone VOQ per server to one VOQ per receiver-side ToR.
SystemC simulation. Because no commercial switches support the proposed VOQ architecture. We resort to a higher
precision simulation, SystemC simulation, to validate theperformance of HierCC under actual devices. The benefit of
SystemC simulation is achieving all the detail steps of the
hardware. Our SystemC simulation is based on the platformof one current commercial switch SystemC code. HierCC
adds additional 2000 lines of C++ program including the
logic of VOQ architecture, credit packets generation andencapsulation, and the first-hop rate control at ToRs.

5

EVALUATION

We compare the performance of HierCC with DCQCN, TIMELY,HPCC and ExpressPass using SystemC and large-scale NS3
simulations. Suggested parameters values are used in thecompared schemes. In HierCC, we set the initial amount ofcredits of each VOQ, C 0 = 10KB, for rapidly pushing out
the first few packets and avoiding the first-RTT time delay at 25Gbps link rate. The expected stable queue length,q 0 = 20KB and β = 0.2. The data bytes that enabled by one

APNet 2021, June 24–25, 2021, Shenzhen, China, China

DCQCN
HierCC

1

2

3

Time(ms)

4

5

1000
800
600
400
200
00

(a) Throughput

6

DCQCN
HierCC

Queue length(KB)

Throughput(Gbps)

25
20
15
10
5
00

X.et al.

123Time(ms)

45(b) Queue length

300
250
200
150
100
50
0

HierCCHPCC
TIMELY
DCQCN
ExpressPass

0.2

0.4

1400
1200
1000
800
600
400
200
0

FCT Slowdown-tailFCT Slowdown-average

Figure 4: Micro-benchmark SystemC simulation.

Load

0.6

0.8

(a) Average FCT Slowdown

HierCCHPCC
TIMELY
DCQCN
ExpressPass

0.2

0.4

Load

0.6

0.8

(b) Tail FCT Slowdown

Figure 5: Large-scale NS3 simulation.
credit: ∆ = 1KB. The VOQ length upper bound Vhiдh = 70KB
and Vlow = 50KB.

5.1

SystemC Simulation

The topology is the same as that in §2.2 but the number ofcore, aggregation, and ToR switches is reduced to 2, 4, 4,
respectively. We run a long-lived flow as the backgroundtraffic and respectively add 15 short 75KB flows at 1ms and3ms as concurrent burst flows. Figure 4(a) and 4(b) show
how throughput and queue length change over time. HierCC
limits VOQs length and aggregates single flows into longlived flows, thus it keeps high link bandwidth utilizationand small queue length while DCQCN builds queue up to
1000KB when concurrent flows start and waste bandwidth
when short flows end.

5.2

NS3 Simulation

The topology is the same as micro-benchmark simulations in§2.2 and we generate realistic data center workload, Cache
Follower [22]. The key results of the simulations are shownin the following. Figure 5(a) and 5(b) show the average and99th percentile FCT with different traffic loads of all themechanisms. HierCC outperforms DCQCN, TIMELY, HPCC
and ExpressPass. HierCC reduces the overall average FCT
by up to 10%, 20%, 50% and 60% compared to HPCC, TIMELY,
ExpressPass and DCQCN. As for the 99th percentile FCT, due
to the fairly credit schedule scheme, HierCC also achieves
lower latency, it reduces up to 20%, 20%, 40% and 50% compared to HPCC, TIMELY, DCQCN and ExpressPass.

RELATED WORKDCQCN [29], TIMELY [17], and HPCC [16] are currently
widely employed RDMA congestion control mechanisms.
HierCC provides more stable and lower latency than themas shown in the Evaluation. Other related works are summarized as follows.
Aggregate flows. Some efforts have been made to improve
the performance of congestion control by aggregating flows
in different contexts. They either share aggregated information to end hosts to facilitate their congestion adjustment[5, 20, 30], or shift queues from the middle of the Internet tothe edge without directly controlling the congestion windowof single flows at end hosts [6]. HierCC differs from themin the way of accurately computing congestion window foreach flow at ToRs and thus achieving ultra-low latency.
Receiver/Swich-driven congestion control. Recently, there
are several proposals aiming to proactively allocate path capacities by sending credits at the receiver [7, 11, 18]. Thesecredits allow sender to send scheduled packets at an ideal rate
and achieve high bandwidth utilization. Besides, RCP[8] iteratively calculates the fair-share rate for each flow at switches
based on the rate assigned last round and local network congestion information. HierCC borrows some ideas from thembut combines them in a new way.
Smaller control loop. Smaller control loop can be achieved
by either hop-by-hop congestion control [3, 25] or fast congestion feedback [23, 27]. Hop-by-hop congestion control
likely to cause HoL blocking and fast congestion feedback
mechanisms still rely on the congestion window adjustment
mechanisms at end hosts.
VOQ-based switch architecture. VOQ-based schemes arealso used in chassis switches (e.g. Arista 7500/7800 series[1])and Stardust[31]. However these two mechanisms convert
packets into cells to be scheduled, while HieCC still leverage
packet switching.

7

CONCLUSION

This paper discusses the shortcomings in existing congestion
control schemes and proposes Hierarchical RDMA Congestion Control (HierCC), which combines aggregating flows,
short control loop and proactive rate allocation to achieve
end-to-end determined small latency. This ingenious design
makes HierCC achieve fast convergence and limited queue
length by using credits between ToR switches and directly
allocate sending rate scheme at the first-hop. We have evaluated the performance of HierCC using SystemC evaluationand large-scale NS3 simulation. Both of the results indicatethat HierCC can significantly mitigate buffer usage and reduce the flow completion time under realistic workload.

HierCC: Hierarchical RDMA Congestion Control

ACKNOWLEDGMENTS
We would like to thank the anonymous reviewers and ourShepherd, Wei Bai, who helped us improve the quality ofthis paper. This work was partly funded by National Natural Science Foundation of China (NSFC) under Grant No.
61872401 and Fok Ying Tung Education Foundation underGrant No. 171059.

REFERENCES[1] [n.d.]. Arista 7500R Switch Architecture. https://www.arista.com/ass
ets/data/pdf/Whitepapers/Arista7500RSwitchArchitectureWP.pdf.
[2] [n.d.]. Cisco Silicon One. https://www.cisco.com/c/en/us/solutions/si
licon-one.html.
[3] [n.d.]. Fibre Channel Industry Association. https://fibrechannel.org/.
[4] 2018. BROADCOM. BCM88690, StrataDNX 10 Tb/s Scalable Switching
Device, Product Brief. https://www.arista.com/assets/data/pdf/White
papers/Arista7500RSwitchArchitectureWP.pdf.
[5] Hari Balakrishnan, Hariharan S Rahul, and Srinivasan Seshan. 1999. An
integrated congestion management architecture for Internet hosts. In
Proceedings of the conference on Applications, technologies, architectures,and protocols for computer communication. 175–187.
[6] Frank Cangialosi, Akshay Narayan, Prateesh Goyal, Radhika Mittal,
Mohammad Alizadeh, and Hari Balakrishnan. 2020. Site-to-Site Internet Traffic Control. arXiv preprint arXiv:2011.01258 (2020).
[7] Inho Cho, Keon Jang, and Dongsu Han. 2017. Credit-scheduled delaybounded congestion control for datacenters. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication.
239–252.
[8] Nandita Dukkipati. 2006. Rcp: Congestion control to make flows complete
quickly. Ph.D. Dissertation. PhD Thesis, Department of ElectricalEngineering, Stanford University.
[9] Peter X Gao, Akshay Narayan, Sagar Karandikar, Joao Carreira, Sangjin
Han, Rachit Agarwal, Sylvia Ratnasamy, and Scott Shenker. 2016. Network requirements for resource disaggregation. In USENIX Symposium
on Operating Systems Design and Implementation (OSDI). 249–264.
[10] Chuanxiong Guo, Haitao Wu, Zhong Deng, Gaurav Soni, Jianxi Ye,
Jitu Padhye, and Marina Lipshteyn. 2016. RDMA over commodity
ethernet at scale. In Proceedings of the 2016 ACM SIGCOMM Conference.
202–215.
[11] Mark Handley, Costin Raiciu, Alexandru Agache, Andrei Voinescu,
Andrew W Moore, Gianni Antichi, and Marcin Wójcik. 2017. Rearchitecting datacenter networks and stacks for low latency and highperformance. In Proceedings of the Conference of the ACM Special Interest Group on Data Communication. 29–42.
[12] Shuihai Hu, Wei Bai, Gaoxiong Zeng, Zilong Wang, Baochen Qiao,
Kai Chen, Kun Tan, and Yi Wang. 2020. Aeolus: A building block
for proactive transport in datacenters. In Proceedings of the Annual
conference of the ACM Special Interest Group on Data Communication on
the applications, technologies, architectures, and protocols for computercommunication. 422–434.
[13] Joseph Izraelevitz, Jian Yang, Lu Zhang, Juno Kim, Xiao Liu, Amirsaman Memaripour, Yun Joon Soh, Zixuan Wang, Yi Xu, Subramanya R
Dulloor, et al. 2019. Basic performance measurements of the intel optane DC persistent memory module. arXiv preprint arXiv:1903.05714(2019).
[14] Dina Katabi, Mark Handley, and Charlie Rohrs. 2002. Congestion
control for high bandwidth-delay product networks. In Proceedingsof the 2002 conference on Applications, technologies, architectures, andprotocols for computer communications. 89–102.

APNet 2021, June 24–25, 2021, Shenzhen, China, China[15] Ana Klimovic, Christos Kozyrakis, Eno Thereska, Binu John, and Sanjeev Kumar. 2016. Flash Storage Disaggregation. In EuroSys.
[16] Yuliang Li, Rui Miao, Hongqiang Harry Liu, Yan Zhuang, Fei Feng,
Lingbo Tang, Zheng Cao, Ming Zhang, Frank Kelly, Mohammad Alizadeh, and et al. 2019. HPCC: High Precision Congestion Control. In
ACM SIGCOMM (Beijing, China). https://doi.org/10.1145/3341302.
3342085[17] Radhika Mittal, Nandita Dukkipati, Emily Blem, Hassan Wassel, Monia
Ghobadi, Amin Vahdat, Yaogong Wang, David Wetherall, David Zats,
et al. 2015. TIMELY: RTT-based Congestion Control for the Datacenter.
In ACM SIGCOMM.
[18] Behnam Montazeri, Yilong Li, Mohammad Alizadeh, and John Ousterhout. 2018. Homa: A receiver-driven low-latency transport protocol
using network priorities. In Proceedings of the 2018 Conference of the
ACM Special Interest Group on Data Communication. 221–235.
[19] Jonathan Perry, Amy Ousterhout, Hari Balakrishnan, Devavrat Shah,and Hans Fugal. 2014. Fastpass: a centralized" zero-queue" datacenter
network. In Proceedings of the 2014 ACM conference on SIGCOMM.
307–318.
[20] Sundararajan Renganathan, Venkata N Padmanabhan, and Akshay Uttama Nambi. 2018. Rethinking Networking for" Five Computers".
In Proceedings of the 17th ACM Workshop on Hot Topics in Networks.
92–98.
[21] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C
Snoeren. 2015. Inside the Social Network’s (datacenter) Network. In
ACM SIGCOMM. 123–137.
[22] Arjun Roy, Hongyi Zeng, Jasmeet Bagga, George Porter, and Alex C
Snoeren. 2015. Inside the social network’s (datacenter) network. In
Proceedings of the 2015 ACM Conference on Special Interest Group on
Data Communication. 123–137.
[23] Ahmed Saeed, Varun Gupta, Prateesh Goyal, Milad Sharif, Rong Pan,
Mostafa Ammar, Ellen Zegura, Keon Jang, Mohammad Alizadeh, Abdul
Kabbani, et al. 2020. Annulus: A Dual Congestion Control Loop for
Datacenter and WAN Traffic Aggregates. In Proceedings of the Annual
conference of the ACM Special Interest Group on Data Communication on
the applications, technologies, architectures, and protocols for computercommunication. 735–749.
[24] Parvin Taheri, Danushka Menikkumbura, Erico Vanini, Sonia Fahmy,
Patrick Eugster, and Tom Edsall. 2020. RoCC: robust congestion control for RDMA. In Proceedings of the 16th International Conference on
emerging Networking EXperiments and Technologies. 17–30.
[25] Chonggang Wang, Kazem Sohraby, and Bo Li. 2005. SenTCP: A hopby-hop congestion control protocol for wireless sensor networks. IEEE
INFOCOM (Poster Paper) (2005).
[26] Jilong Xue, Youshan Miao, Cheng Chen, Ming Wu, Lintao Zhang, andLidong Zhou. 2019. Fast distributed deep learning over RDMA. In
Proceedings of the Fourteenth EuroSys Conference 2019. 1–14.
[27] David Zats, Anand Padmanabha Iyer, Ganesh Ananthanarayanan,
Rachit Agarwal, Randy Katz, Ion Stoica, and Amin Vahdat. 2015. Fastlane: making short flows shorter with agile drop notification. In Proceedings of the Sixth ACM Symposium on Cloud Computing. 84–96.
[28] Jiao Zhang, Fengyuan Ren, Ran Shu, and Peng Cheng. 2016. TFC: token
flow control in data center networks. In Proceedings of the Eleventh
European Conference on Computer Systems. 1–14.
[29] Yibo Zhu, Haggai Eran, Daniel Firestone, Chuanxiong Guo, Marina
Lipshteyn, Yehonatan Liron, Jitendra Padhye, Shachar Raindel, Mohamad Haj Yahia, and Ming Zhang. 2015. Congestion Control for
Large-Scale RDMA Deployments. SIGCOMM Comput. Commun. Rev. 4(2015). https://doi.org/10.1145/2829988.2787484[30] Danyang Zhuo, Qiao Zhang, Vincent Liu, Arvind Krishnamurthy, andThomas Anderson. 2016. Rack-level congestion control. In Proceedingsof the 15th ACM Workshop on Hot Topics in Networks. 148–154.

APNet 2021, June 24–25, 2021, Shenzhen, China, China[31] Noa Zilberman, Gabi Bracha, and Golan Schzukin. 2019. Stardust:
Divide and conquer in the data center network. In 16th {USENIX}

X.et al.
Symposium on Networked Systems Design and Implementation ({NSDI}
19). 141–160.