IEEE 802.1 July 2022 Plenary Session

Source Flow Control Design: Caching
P802.1Qdw contribution

Jeremias Blendin
Contributors: Jeongkeun "JK" Lee, Yanfang Le, Paul Congdon

SFC High Level Concept

Today: 802.1Qbb - Priority-based Flow Control (PFC)
PFC
HoLB

PFC

PFC

PFC

HoLB

• Signal from switch directly to traffic source:
per-flow pausing

PFC§ Source Flow Control

Congestion

PFC• Removes head-of-line blocking from network

Proposed: Source Flow Control (SFC)
SFCM• Simplify deployments compared to PFC• Does not require complex buffer tuningCongestion• Completely remove risk of deadlocks

SFC Message(SFCM)

Congested Flow
Vic0m FlowIEEE 802.1 September 2022 Interim

SFCM

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

2SFC w/ ToR Proxy (SFC-P)§ SFC with ToR Proxy

Source Flow Control (w/ ToR Proxy)
SFCM• Works with today's RDMA NICs
• SFC proxy converts SFC message toPFC frame at sender ToR

GenerateCongestion

InterceptPFC

SFC Message (SFCM)• Removes congestion from network• HolB possible at sender NICs but not in switchesSFCM

Congested Flow
Victim FlowIEEE 802.1 September 2022 Interim

Figure source: https://mentor.ieee.org/802.1/dcn/21/1-21-0068-01-ICne.pdf

3Design Discussion

IEEE 802.1 September 2022 Interim4

Topic 3: Contents of SFCM
What needs to be in the SFCM? Should it include Qau ‘quantized’ parameters?
Explanation/Solution:• Qau specifies ‘quantized’ parameter Fb. CNM message carries Fb to host as input of rate calculation.

(From 802.1Q -2018 30.2.1 CP algorithm)

Focus of this• SFCM is sent to the sending host and is interpreted as if a PFC frame was received, F is not needed.
discussion• SFC proxy mode generates a PFC frame and does not need Fb. Pause time is needed.

b• Source IP address of offending flow is needed to generate SFCM

• Offending flow information is needed so source can map SFCM to appropriate traffic class. This includes DSCP
• A congestion locator such as Topology Recognition level to identify ‘incast’ congestion verses ‘in-network’ congestion.
• An optional PTP timestamp when the message is sent to assist in pause duration adjustments at the source.
IEEE 802.1 September 2022 Interim

Source: SFC Design Team: “SFC Design Team Topics”, IEEE March 2022

5

Topic 4: Identifying the source priority/TC to pause
The priority/TC used to send the packet at the source may be different than thepriority/TC received at the congestion point. Which priority/TC to pause?

Explanation/Solution:• SFCM includes information to identify the flow which should be paused,
as well as pause time.
• Because of the provided flow information in the SFCM, the source knows
which queue (priority) needs to be paused.
• PFC can be generated to the source accordingly.
IEEE 802.1 September 2022 Interim

Source: SFC Design Team: “SFC Design Team Topics”, IEEE March 2022

6

Terminology used in this Slide DeckCaching andSignaling Point

SFC Message(SFCM)

Receiver

SenderReaction pointSFC
SFM Reception SFM Reception
point SFC-P
point SFC

Congestion and
Signaling Point

Terminology based on QCN (802.1Qau) andSFC Design Team: “SFC Design Team Topics”, IEEE March 2022
IEEE 802.1 September 2022 Interim

7

SFC Message Contents: What to Pause?
§ Baseline• Use first X bytes of original packet• SFC: Reaction point Sender NIC• Identify the flow to pause• How? Match original packet fields• SFC-P: Reaction point Sender ToR• Identify the TC to pause• How? Use DSCP value from original packet header• Simple, yet effective• Do not consider caching (details later)
IEEE 802.1 September 2022 Interim

8Baseline SFC Message Contents
SFC Header
version,header length,
etc

Pause
duration[us]

x bytes oforiginal
packetheaderTCP
IP…
Ethernet
IEEE 802.1 September 2022 Interim

9

SFC Caching

IEEE 802.1 September 2022 Interim

10Caching Overview§ For incast scenarios, caching is important• For some scenarios caching might not be possible§ Caching points (details on next slides)• Sender ToR and Sender NIC• Caching logic should not be part of the standard§ Use Congestion Point Locators• Specify traffic patterns to pause• The original packet header might not be a good fit for all cases• When tunneling is used: caching point needs to parse header stack• IPv6: destination host might have a /64 prefix assigned• Multiple DSCP values might map to TC/congested queue

IEEE 802.1 September 2022 Interim

11

SFCM Caching Opportunities
Sender
NICRDMAEngine

SFC Engine
w/ Cache

RDMA qp1

Host f1
Host f2

5⃣
RDMA qp2
Pause
queue
6⃣ New ﬂow

IEEE 802.1 September 2022 Interim

SwitchSFC
Cache

4⃣ SFCM
Port /
Link

7⃣❌ Pausequeue

Port /
Link

1⃣ Incoming SFCM
3⃣
SFCM
fromCachePort /
Link

Port /
Link

2⃣ New ﬂow
Port /Link
12

Caching: SFC and SFC-P§ Caching andSFC Proxy areseparate concepts

Caching andSignaling Point

SFC Message(SFCM)

Receiver

SenderReaction point
SFC• The table columns
show what eachactor does in SFC and SFC-PCongestion
Point

SFM Reception
point SFC

Sender ToR

SFM Reception
point SFC-P

Sender NIC

Congestion andSignaling Point

Sender ToR
Cache

Sender NIC
Cache

SFC

Sends SFCM

N/A

Process SFCM

Trigger SFCM

Trigger NICinternal signal

SFC-P

Sends SFCM

Convert SFCM
to PFC frame

Process PFC
frame

Trigger PFC
frame

N/A

IEEE 802.1 September 2022 Interim13

Congestion Point Locators§ Specification of congested queue• Enable senders to identify traffic
going to the congestion point
within the pause period

Spine switch
Sender ToRUplink§ Last-hop case• Congestion point is part ofall paths to the receiver• Covers incast use cases§ Other cases• Congestion point is only part of a subsetof paths to the receiverIEEE 802.1 September 2022 Interim

Multi homing

Last-hopcase
Simple case14

Congestion Point Locator: Last-hop case
§ From original packet header: Use inner destination IP and DSCP value§ Specify explicitly in SFCM header• Port identification: Destination prefix of receiver• Queue identification• List of PCP/DSCP values that map to the queue on the congested switch• Complex header format (list with up to 64 6bit values)• PCP/DSCP to TC mappings might be different on different switches• No TC mapping synchronization between reaction and signaling point required• TC as is used by PFC• Simple: can use 8bit one hot encoding• Requires consistent traffic to TC mappings in reaction point and signaling point
IEEE 802.1 September 2022 Interim

15

Our Thoughts SFC Message Contents
SFC Message

Possible options

version,Pauseheaderduration
length, etc[us]Cache: Cache: Cache:Useprefix1 prefix2
packetTC
DSCP
header bitmap values

x bytesoforiginal
packetheader

TCPIP…
Ethernet
IEEE 802.1 September 2022 Interim16

17