IEEE 802.1 July 2022 Plenary Session

Source Flow Control Simulation Results
Jeremias Blendin
Contributors: Jeongkeun "JK" Lee, Yanfang Le, Pedro Yebenes Segura, Paul Congdon

Agenda§ SFC Introduction§ Simulation Overview§ Comparing SFC with PFC§ Comparing Workloads§ SFC Parameter Sensitivity

IEEE 802.1 July 2022 Plenary

2

SFC High Level Concept

Today: 802.1Qbb - Priority-based Flow Control (PFC)
PFC
HoLB

PFCPFC• Signal from switch directly to traffic source:
per-flow pausing

PFC

HoLBPFC§ Source Flow ControlCongestion• Removes head-of-line blocking from network
PFC• Simplify deployments compared to PFC• Does not require complex buffer tuning

Proposed: Source Flow Control (SFC)
SFCM• Completely remove risk of deadlocks§ How is SFC triggered in our implementation?
• Trigger on egress queue depth threshold

Congestion
SFC Message(SFCM)• Sender pause time = expected drain time
to target queue depth• Not necessarily part of the standard (see earlier discussion
https://www.ieee802.org/1/files/public/docs2022/newcongdon-SFC-design-topics-0622-v01.pdf)

IEEE 802.1 July 2022 Plenary

Congested Flow
Victim Flow

SFCM

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

3SFC w/ ToR Proxy§ SFC with ToR Proxy

Source Flow Control (w/ ToR Proxy)
SFCM• Works with today's RDMA NICs
• SFC proxy converts SFC message toPFC frame at sender ToR

GenerateCongestion

Intercept

PFC

SFC Message (SFCM)• Removes congestion from network• HolB possible at sender NICs but not in switches
SFCM

Congested Flow
Victim FlowIEEE 802.1 July 2022 Plenary

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

4Simulation Overview

IEEE 802.1 July 2022 Plenary

5Simulation: Goals§ Show behavior of SFC• Application performance§ Metrics• Application performance: Flow Completion Time (FCT) slowdown• Compare the measured FCT with FCT of flow that is transmitted in unloaded network§ Methodology• Evaluate flow control mechanisms by measuring effects of incast traffic on
non-incast traffic (background traffic) in the network• Transport protocol: RDMA with DCQCN• State-of-the-art flow control in modern RDMA NICs
• Use variant with “initial window” mechanism (DCQCN+W) introduced in HPCC paper• Follow state-of-the-art approach:
Li, Yuliang, et al. "HPCC: High precision congestion control." ACM SIGCOMM 2019.
IEEE 802.1 July 2022 Plenary

6Simulation Setup§ Simulation Software• NS3: Based on the open-source code published for theHPCC paper

https://github.com/alibaba-edu/High-Precision-Congestion-Control§ Network topology (Follow HPCC)• 3-tier fat-tree (100/400GbE)• 320 nodes, 56 switches• Full bisection bandwidth• 12us round-trip time (RTT)§ PFC, DCQCN, SFC parameters inbackup slides• Note: PFC + DCQCN is sensitive to tuning(workload-specific)
IEEE 802.1 July 2022 Plenary

7Comparing SFC with PFC

IEEE 802.1 July 2022 Plenary8Traffic Load Configuration§ Background traffic§ In-cast traffic• “normal” application workload on the network• 100:1 incast• 320 hosts• Message size 256 KB• Traffic based on Google RPC workload• The incast traffic load is 10% of thenetwork capacity• Source: Montazeri, Behnam, et al. "Homa: A receiver-driven low-latencytransport protocol using network priorities."ACM SIGCOMM 2018.

• Load factor 40%• Trace is 16x longer than the one presented atthe IEEE 802.1 May 2022 Interim meeting

IEEE 802.1 July 2022 Plenary

9Flow Completion Time Results• Significant
improvementsof SFC and SFC
w/ Proxy forsmall messages• SFC
outperforms
PFC in allpercentiles• SFC w/ Proxy
performs SFC
for smallmessage sizes
and on par with
PFC for large
messages

Slowdown compared to reference [log10]§ Key findings

Workload: GoogleRPC 40%,
incast 100:1

Workload: GoogleRPC 40%,
incast 100:1

Workload: GoogleRPC 40%,
incast 100:1

Percentile: 50

Percentile: 99

Percentile: 99.99100

101100 B

10 kB

1 MB

Flow control

IEEE 802.1 July 2022 Plenary

100 B
10 kB
1 MBMessage size [bytes, log10]
PFC

SFC (2>1)

100 B

10 kB

1 MB

SFC w/ Proxy (2>1)

Trigger >Target
in BDP

10

Comparing Workloads

IEEE 802.1 July 2022 Plenary

11

Investigating a wider range of Workloads§ Background traffic§ In-cast traffic• 320 hosts• 100:1 incast• Workloads• Message size 256 KB• DCTCP

Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra
Padhye, Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, andMurari Sridharan. Data center tcp (dctcp). In Proceedings of theACM SIGCOMM 2010 Conference, SIGCOMM ’10, 2010.

• The incast traffic load is 10% of thenetwork capacity• Load factor: 40%• FacebookHadoop

ArjunRoy,HongyiZeng,JasmeetBagga,GeorgePorter, and Alex C.
Snoeren. Inside the social network’s (dat- acenter) network. In
Proceedings of the 2015 ACM Conference on Special Interest
Group on Data Commu- nication, SIGCOMM ’15, 2015.

• Load factors: 40%, 80%
IEEE 802.1 July 2022 Plenary

12Flow Completion Time
• Significantimprovements ofSFC and SFC w/
Proxy for smallmessages• SFC outperforms
PFC significantly
in all percentiles
and all message
sizes• The difference
between SFC andSFC w/ Proxy forsmall messages
sizes is lesspronounced thanin the GoogleRPC
workload

Slowdown compared to reference [log10]§ Key findings

DCTCP 40%,
incast 100:1

FacebookHadoop 40%,
incast 100:1

FacebookHadoop 80%,incast 100:1

Percentile: 99.99

Percentile: 99.99

Percentile: 99.99

100

101
1 kB

100 kB

10 MB

Flow control

IEEE 802.1 July 2022 Plenary

1 kB
100 kB
10 MBMessage size [bytes, log10]
PFCSFC (2>1)

1 kB

100 kB

10 MB

SFC w/ Proxy (2>1)

13

SFC Parameter Sensitivity

IEEE 802.1 July 2022 Plenary

14

Investigating SFC sensitivity to trigger and target
parameters§ Background traffic§ SFC Parameters• 320 hosts• Trigger threshold and target queue depth• Workload: Google RPC• BDP: 12us RTT * 100GbE = 150KB• Load factor 40%• Combinations: Trigger>Target• 1BDP > 0.5BDP§ In-cast traffic• 2BDP > 1BDP

• 100:1 incast• 4BDP > 2BDP

• Message size 256 KB• The incast traffic load is 10% of the network
capacityIEEE 802.1 July 2022 Plenary

15

SFC Parameter Results
GoogleRPC 40%,
incast 100:1

Percentile: 50

Percentile: 99

Percentile: 99.99

100101SFC w/ Proxy• Smooth
response in
FCT outcome
for different
parameter
settings

GoogleRPC 40%,
incast 100:1SFC• SFC and SFC w/
Proxy behave
consistently
with different
parameter
settings

Slowdown compared to reference [log10]§ Key findings

GoogleRPC 40%,
incast 100:1

100101100 B

10 kB

Flow control

IEEE 802.1 July 2022 Plenary

1 MB100 B
10 kB
1 MBMessage size [bytes, log10]

SFC (1>0.5)
SFC (2>1)

SFC (4>2)
SFC w/ Proxy (1>0.5)

100 B

10 kB

1 MBSFC w/ Proxy (2>1)
SFC w/ Proxy (4>2)

Trigger >Targetin BDP

16

Conclusion

IEEE 802.1 July 2022 Plenary

17Conclusion§ SFC and SFC w/ Proxy performance• SFC can significantly reduce FCT for background traffic• Not surprisingly, workloads have a significant impact on performance improvements• FCT performance order is consistent: SFC > SFC w/ Proxy > PFC• SFC w/ Proxy: brown field approach that improves PFC performance
without its operational downsides• Both SFC variants behaved consistently and good-natured to parameter changes§ Future work
• Systems results

IEEE 802.1 July 2022 Plenary

18

19

Benefits Simulation Settings

IEEE 802.1 July 2022 Plenary

20

System Configuration in FabSim-X§ Network• 3-tier fat-tree• 320 nodes to 20 ToR sw (1 link)• 20 ToR sw to 20 agg sw (1 links)• 20 agg sw to 16 core sw (1 links)• Switch radix: 20• Link delay: 0.5 us• Link speed SW to SW: 400 Gbps
• Link speed NIC to SW: 100 Gbps
• MTU: 1024 B• RTT 12 usIEEE 802.1 July 2022 Plenary

21

Switch Configuration Parameters• SFC: Shared buffer• SFC configuration• Ingress pool size: disable accounting (200 MB)• Suppression period: 6 us (1/2 RTT)

• Egress pool size: 16 MB• Destination cache: ToR• Ingress guaranteed per port: 4 KB
• Egress guaranteed per port: 4 KB
• Sharing mechanism: dynamic thresholding• PFC: Dynamically shared
Ingress Buffer (HPCC paper)• Buffer: 13 MB pool + 3MB headroom• Ingress coefficient: 2• Dynamic sharing coefficient: 0.125• Egress coefficient: 2• No egress drops• No ingress drops (set it to a very high value),
but we can have egress drops

IEEE 802.1 July 2022 Plenary

22

Congestion control configurations§ DCQCN• Using parameters as defined in: Li, Yuliang, et al. "HPCC:
High precision congestion control." ACM SIGCOMM 2019.
• Fast recovery steps: 1• Gain: 0.00390625• Rate increase timer: 900• Alpha timer: 1 us
• CNP period: 4 us
• AI: 0.05 Gbps
• Hyper AI: 0.1 Gbps
• Window: 15us• ECN threshold: 150 KB (1 BDP)IEEE 802.1 July 2022 Plenary23