IEEE 802.1 May 2022 Interim Session

Source Flow Control Simulation Results
Fairness and Performance
Jeremias Blendin
Contributors: Jeongkeun "JK" Lee, Yanfang Le, Pedro Yebenes Segura, Paul Congdon

Agenda§ SFC Introduction§ Simulation Overview§ SFC Benefits§ SFC Fairness

IEEE 802.1 May 2022 Interim Session

2

Source Flow Control§ Background: Future 802.1 Congestion Management Tools• 802.1Qbb - Priority-based Flow Control (PFC)• Hop-by-hop flow control• P802.1Qcz - Congestion Isolation (CI)• Improve PFC by isolating congested queues and reduce hop-by-hop head-of-line blocking§ This talk: Source Flow Control (SFC)• Signal from switch directly to traffic source• Remove head-of-line blocking from network• SFC w/ Proxy design to accelerate deployment• Does not require complex buffer tuning
IEEE 802.1 May 2022 Interim Session

3Source Flow Control (SFC) High Level Concept
Proposed: Source Flow Control

Today: 802.1Qbb - Priority-based Flow Control (PFC)
PFC

SFCM

Congested Flow
Victim Flow

HoLB

P FC
Congestion

Congestion

PFC

SFC Message (SFCM)

PFHoLBCPFC

SFCMPFC§ Operational concerns§ Benefits over PFC• Head-of-Line blocking• Use local switch telemetry to trigger back-to-sender signaling• Congestion spreading• Edge-to-edge FC signaling using L3 message• Buffer Bloat, increasing latency• Removes head-of-line blocking completely from the network• Increased jitter reducing throughput• SFC signaling directly to transport protocol end-point (per-flow)• Deadlocks with some implementations• Works with many types of transport protocols (RoCEv2, TCP, UDP)

IEEE 802.1 May 2022 Interim Session

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

4Source Flow Control (SFC) w/ ToR Proxy§ SFC with ToR Proxy

Source Flow Control (w/ ToR Proxy)
SFCM• SFC proxy converts SFC message to PFC frame atsender ToR• Works with today's RDMA NICs
• Removes congestion fromnetwork switches

Generate
Intercept

PFCCongestion

SFC Message (SFCM)• Only small chance for head-of-line blocking at sender NIC

SFCM

Congested Flow
Victim FlowIEEE 802.1 May 2022 Interim Session

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

5Simulation Overview

IEEE 802.1 May 2022 Interim Session

6

Simulation: Goals§ Show benefits of SFC• Increase application performance• Reduce of in-network buffering• Does not affect fairness§ Metrics• App perf: Flow Completion Time (FCT)• In-network buffering: Switch buffer occupancy• Fairness: Per-flow link capacity share

IEEE 802.1 May 2022 Interim Session

7

Simulation Setup§ Simulation Software• Fabsim-X (Intel proprietary)•Musleh, Malek, et al. "Fabsim-X: A simulation framework for the analysis oflarge-scale topologies and congestion control protocols in data center networks."
IEEE MASCOTS 2020.

•

Note: we observed comparable results on NS3§ Network topology• 3-tier fat-tree (100/400GbE)• 320 nodes, 56 switches• Full bisection bandwidth§ PFC, DCQCN, SFC parameters inbackup slides• Note: PFC + DCQCN is sensitive to tuning(workload-specific)• PFC/DCQCN parameters are selected
to ensure losslessness
IEEE 802.1 May 2022 Interim Session8

Traffic Load Configuration§ Network protocol• RDMA with DCQCN• State-of-the-art flow control in modern
RDMA NICs• Use variant with “initial window” mechanism• Source: Li, Yuliang, et al. "HPCC: High precisioncongestion control." ACM SIGCOMM 2019.

§ Background traffic§ In-cast traffic• 120:1 incast• Message size 256 KB• The incast traffic load is 8% of the networkcapacity• Similar approach to: Li, Yuliang, et al. "HPCC: High precisioncongestion control." ACM SIGCOMM 2019.

• 320 hosts• Traffic based on Google RPC workload• Source: Montazeri, Behnam, et al. "Homa: A receiver-driven low-latencytransport protocol using network priorities."
ACM SIGCOMM 2018.

• Load factor 50%
IEEE 802.1 May 2022 Interim Session

9SFC Benefits

IEEE 802.1 May 2022 Interim Session

10Results: Background Traffic Performance
Background FCT - messages < 10 KB
1600

1800

FCT
reduction
>60x

1200

1200

3286

FCT
reduction
~2x

3000

1209
2500

us

us

1000

800

3500

FCT
reduction
~10x

1400

1000

us

3562

1600

1034

4000

1705

1394

1400

Background FCT - messages > 1 MB

Background FCT - messages > 10 KB
and < 1 MB

2000

mean_

1798

p95_

1577

800
600

1455

1500

p99_

600
400

372

400

1000

273

221
200

182

200
70PFC

12 22SFCDCQCN

IEEE 802.1 May 2022 Interim Session

7817 4315SFC w/ Proxy

117545

474

500

247

226

2800PFCSFC
DCQCN

SFC w/ Proxy

PFC

SFC

SFC w/ Proxy

DCQCN
11

Results: Incast Performance and Queue Depth
Incast FCT - 256 KB messages

Queue depth

3000

16000

2500

13634

14000

2483
2304

2264

2377

2349

2263

Queuedepth
reduction
>20x

12000

2000

mean_

1500

KB

KB

10000

mean_

8000

p95_
1020

1008

1000

993

p99_

p95_

6218

p99_

6000

4000
500
2000
795
0280PFC

SFC
DCQCN

IEEE 802.1 May 2022 Interim Session

SFC w/ Proxy

PFC

91585

SFC

2173 280

SFC w/ Proxy

DCQCN

12Impact on Fairness

IEEE 802.1 May 2022 Interim Session

13

Impact on Fairness: Overview§ Goal• Show that SFC’s effect on fairness§ Approach• Use PFC as baseline• Staggered start/stop to observe flow-addition and flow-removal behavior
on congested link•

As suggested in: Perry, Jonathan, et al. "Fastpass: A centralized" zero-queue"
datacenter network." ACM SIGCOMM 2014.

• Use long-running, similar-sized flows§ When flow sizes are mixed, fairness index is not clearly defined• FCT partially reflects fairness, as worsend fairness will push up tail latency§ Scenarios• 4:1 incast, 100GbE• Intra-switch: 4 senders, 1 receivers connected to same switch• Inter-switch: Intra scenario with 2 senders connected to remote ToR switch

IEEE 802.1 May 2022 Interim Session

14

SFC Fairness Comparison with PFC§ SFC does not hurtfairness provided byDCQCN

IEEE 802.1 May 2022 Interim Session

15

Conclusion§ Benefits compared to PFC• SFC can significantly reduce FCT for background traffic• SFC can significantly reduce queueing in the network§ No negative side-effects• SFC does not negatively affect the incast FCT• SFC does not negatively affect fairness§ Future work
• Bigger scale and higher RTT simulations• Systems results
IEEE 802.1 May 2022 Interim Session

16

17

Benefits Simulation Settings

IEEE 802.1 May 2022 Interim Session

18

System Configuration in FabSim-X§ Network• 3-tier fat-tree• 320 nodes to 20 ToR sw (1 link)• 20 ToR sw to 20 agg sw (1 links)• 20 agg sw to 16 core sw (1 links)• Switch radix: 20• Link delay: 0.5 us• Link speed SW to SW: 400 Gbps
• Link speed NIC to SW: 100 Gbps
• MTU: 1024 B• RTT 12 usIEEE 802.1 May 2022 Interim Session

19

Switch Configuration Parameters• SFC: Shared buffer• SFC configuration• Ingress pool size: disable accounting (200 MB)• Trigger threshold: 100 KB (2/3 BDP)

• Egress pool size: 16 MB• Target threshold: 50 KB (1/3 BDP)

• Ingress guaranteed per port: 50 KB• Suppression period: 6 us (1/2 RTT)

• Egress guaranteed per port: 50 KB• Destination cache: ToR• Sharing mechanism: dynamic thresholding• PFC: Static Ingress Buffer• Ingress coefficient: 1• Total buffer 16 MB / 20 ports = ~800 KB• Egress coefficient: 1• PFC threshold 650 KB• No ingress drops (set it to a very high value),
but we can have egress drops

IEEE 802.1 May 2022 Interim Session• No egress drops20Congestion control configurations§ DCQCN• Fast recovery steps: 1• Gain: 0.0009813• Byte counter: 2097152• Timer: 4 us
• Alpha timer: 5 us
• AI: 0.0125• Hyper AI: 0.025• CNP period: 4 us
• Window: 15us
• ECN threshold: 50 KB (1/3 BDP)IEEE 802.1 May 2022 Interim Session

21

Fairness Simulation Settings

IEEE 802.1 May 2022 Interim Session

22

Switch Configuration Parameters• Shared buffer: Both

• SFC configuration• Egress pool size: 16 MB• Trigger threshold: 100 KB• Ingress guaranteed per port: 50 KB• Target threshold: 64 KB• Egress guaranteed per port: 50 KB• Sharing mechanism: dynamic thresholding• Ingress coefficient: 0.5• Egress coefficient: 2.0• Shared buffer: SFC• Suppression period: 6 us
• Destination cache: Only in ToR• Other settings are the same as theBenefits Simulation Setup• Ingress pool size: disable (200 MB)• Shared buffer: PFC• Ingress pool size: 9 MBIEEE 802.1 May 2022 Interim Session

23