Adaptive PFC Headroom
Lily Lv (Huawei)

Outline• PFC Recap : Accurate ‘Headroom’ is Important for PFC• Complexity of Headroom Calculation• What We Have in IEEE 802• What is Still Missing• Proposal for Adaptive PFC Headroom• Next StepsRecap: PFC Concept• Priority based Flow Control (PFC) is defined in Clause 36 of IEEE Std 802.1Q-2018• Mainly used in data center networks in order to avoid packet loss due to congestion.
• “PFC allows link flow control to be performed on a per-priority basis. In particular, PFC is used to inhibittransmission of data frames associated with one or more priorities for a specified period of time. PFC can be
enabled for some priorities on the link and disabled for others.” (Std 802.1Q-2018)• One example of PFC• XOFF threshold which invokes PFC is set on each PFC enabled priority• Priority 6 reaches threshold XOFF• PFC pause frame is triggered and sent upstream• Upstream priority 6 transmission is stopped
PFC enabled

Station 2

Station 1

Priority 0

Priority 0

Priority 1

Priority 1

Priority 2

Priority 2

Priority 3

Priority 3

Priority 4Priority 4

Priority 5
Priority 6 stop
Priority 7

Priority 5
PFC Pause

Priority 6
Priority 7

XOFF XOFF

Recap: PFC Delay and Headroom• There is a time delay between PFC
invocation on sender and pause action onreceiver.

• This PFC delay requires the PFC sender(station 1 in figure) reserve buffer toabsorb in-flight packets.
• The reserved buffer is also called‘headroom’.

Accurate ‘Headroom’ is Important for PFC• PFC headroom and bridge buffers share the same buffer
pool in many implementations.

XOFF• When PFC is not invoked, bridge buffer is used to absorb
normal traffic burst

Bridgebuffer• When PFC is invoked, buffers as PFC Headroom is used toabsorb inflight packets
Bridge buffer is used to absorb burst traffic, to avoid packet loss.
PFC• XOFF threshold setting relates to headroom.

headroom
bufferStop

Bridgebuffer• If XOFF threshold is too high ( less headroom )• packet drop may happen, not ‘lossless’ anymore.

PFC must reserve headroom buffer to absorb inflight packet, toavoid packet loss.

• If XOFF threshold is too low ( more headroom )• traffic is suspended unnecessarily, low network bandwidth utilization.
• Buffer resource is wasted.

• By calculating headroom, optimal XOFF threshold could be set.

Complexity of Headroom Calculation• Delay value calculation:• PFC transmission delay need consider maximum
length frame as worst case, as well as PFC frame
itself.

• Interface delay and higher layer delay are vendor
implementation dependent• 802.3 defines maximum value of such delays, however,vendors can do much better than that.

• Medium delay is port speed, media and distancedependent
Figure N-2—Delay model (802.1Q-2018)

Complexity of Headroom Calculation• One example from 802.1Q-2018 Annex N, assuming 100 meterscat6 cable, 10G BASE-T with maximum interface delay andhigher layer delay.
DV = 2 × (Max Frame) + (PFC Frame) + 2 × (Cable Delay) + (TXds1 + RXds2) + (TXds2 +RXds1) + HDs2
DV = 2 × (16 160) + (672) + 2 × (5556) + (25 600 + 12 288) + (25 600 + 12 288) + 6144=
126 024 bit times = 15.5KB* When it comes to 100G or above, or when cable length increases, such as data centerinterconnection, cable delay will be significant.

• Furthermore, implementation dependent internal buffer
fragmentation should be considered when calculating
headroom.
Figure N-3—Worst-case delay (802.1Q-2018)• Buffer to store the packet is usually allocated chunk by chunk, not bytestream FIFOs, e.g. 160 bytes as smallest chunkCurrent PFC Headroom Reservation in Network Management is Not
Efficient• Usually, network engineers consider headroom
during network deployment or network changes.
• One common way is to use default value from vendor.
However, this rarely matches the real environment.

10G, 5m@port1

25G,500m@Port2

100G,50km@Port3

xG,xm@all ports

headroom buffer

No packet loss

Packet loss

bridge buffer

Packet loss

Enough bridge buffer,burst resilient• Variable distance impacting ‘cable delay’, especially in long
distance Data Center Interconnect (DCI) scenario

headroom
bufferbridge buffer• Variable implementation dependent hardware processing
impacting Interface Delay, Higher Layer Delay.

No packet loss

No packet loss

No packet loss

Less bridge buffer, notburst resilient, may be inlow throughput

ID + HD ( bit times )Cable delay (bit times)
100G Base-R

10m

5000 (0.6KB)

10km

5 000 000 (625KB)

0.6KB for 10m estimation error (DCN case); 625KB for 10km estimation error (DCI case)

100G Base-R

802.3 max value

132 608

Test value

100 000

Default settings may increase actual needs by 33%• Otherwise, manual calculation, configuration and test are required based on hop by hop distance,transmission rate, etc. That is time consuming.

Network Management Prefers ‘Plug-and-Play’As a network management engineer, I want to

10G, 5m@port1

25G,500m@Port2

100G,50km@Port3• Place the switch wherever it is needed and assure losslessbehavior.
• Not worry about an improper configuration ( e.g default values)
which might cause performance issue.

headroom buffer

No packet loss

Packet loss

Packet loss

Bridge buffer

Enough bridge buffer,burst resilient• Release me from learning complex operations (tools,commands, parameters, etc.) on different vendors’ equipment,
requiring me to read hundreds of pages of instructions.

• Shorten the network BIS(Bring into Service) time and reduce
OPEX

xG,xm@all ports

headroom
bufferbridge buffer

No packet loss

No packet loss

No packet loss

Less bridge buffer, notburst resilient, may be in
low throughput

headroom
buffer

If the headroom setting is automatically adapted toenvironment, like a ‘plug and play’ feature, thenetwork manager’s objectives can be met.

bridge buffer

No packet loss

No packet loss

No packet loss

Adaptive buffer size,
burst resilient and nopacket loss

What We Have in IEEE 802
 802.3 90 Ethernet
support for timesynchronization
protocols--Data delaymeasurement

 802.1AS 11.1.2“Propagation delaymeasurement”

 802.1Qcc 12.32.2“Propagation delay ”

Measure tx/rx data path delay to support time synchronization
Data path delay is between xMII and MDI.
* Note: Current 802.3 timestamping is done at the xMII, 802.3cx propose to movethe timestamp to MDI to improve time synch accuracy.

Use std 1588-2019 two-step PTP mechanism on a full-duplex
point-to-point PTP link

The txPropagationDelay attribute is typicallymeasured using a time synchronization protocol，e.g. 802.1AS

What is Missing ……
• The current set of standards are developed for time synchronization• Data center networks seldom activate time synchronization.
• No description of usage for PFC headroom calculation.
• Time synchronization has different preference of timestamp points from headroom calculation.
802.1 ASPFC Headroom
Time sync preferred
timestamp point

PFC Headroom preferred
timestamp point

Proposal for Adaptive PFC Headroom (1/4)
Station 2

Station 1• Phase 1: Capability notification• Augment DCBX by extending PFC configuration TLV

PFC HDRM capability notification• DCBX uses LLDP with new PFC configuration TLV to exchangecapability• If both support PFC HDRM, initiate PFC HDRM Measurement
Request, otherwise, stop the procedure.

PFC HDRM Measurement Request(Request frame contains sending timestamp T1)

PFC HDRM Measurement Response(Response frame set ACK to 1. T2 and T3 arecontained. T2 is receiving timestamp, T3 issending timestamp )Set receiving
timestamp T4.

HDRM
SupportReserved

1 bit

1 bitExample

Proposal for Adaptive PFC Headroom (2/4)• Phase 2: Delay Measurement
Station 2Station 1• Measurement request is sent from station 1 to station 2 with sendingtimestamp T1• Measurement response is sent from station 2 to station 2 withreceiving timestamp T2 and sending timestamp T3

PFC HDRM capability notification• Station 1 set receiving timestamp T4T1

• Measurement request and response frame is a new MAC control
frame

PFC HDRM Measurement Request(Request frame contains sending timestamp T1)

T2
PFC HDRM Measurement Response(Response frame set ACK to 1. T2 and T3 arecontained. T2 is receiving timestamp, T3 issending timestamp )identify Measurement
frameT30x0000：measurement request
0x0001: measurement response

Set receiving
timestamp T4.

T4
Packet sequence number

PFC frame format

Measurement frame format

ExampleProposal for Adaptive PFC Headroom (3/4)
Example of Adaptive headroom

802.1Q PFC

Station 2

Station 1
Last bit ofMreq(measurementrequest) frame passed
to MAC service
Last bit of Mreq frame
sent into the wire

③T1
TXds1 BTs④
bit of Mreq frame⑤ Lastreceived from the wire

RXds2 BTs+HDs2 BTs
Generating of Mresp should
be removed from PFC delay

TXds2 BTs
Last bit of Mresp fromstation 2 received fromthe wire
Last bit of Mresp fromstation 2 received.

DV = 2*(Max Frame) + (PFC Frame) + 2*(Cable Delay) + TXds1 + RXds2 + HDs2 +TXds2 + RXds1

T2 ⑥ Mresp(measurement response)
frame M_CONTROL_request is
invokedT3

bit of Mresp frame⑦ Lastpassed to MAC service

bit of Mresp frame sent
⑨ Lastinto the wire⑩⑪T4

RXds1 BTs

X = (T4-T1- (T3-T2) ) * Speed = 2*(Cable Delay) + TXds1 + RXds2 + HDs2 + TXds2 + RXds1
DV = 2*(Max Frame) + (PFC Frame) + X

Proposal for Adaptive PFC Headroom (4/4)
Example• Phase 3: Headroom calculation• X = Port speed * (T4-T1-(T3-T2))

Measurement

Measurement

T4T2• DV = X + 2*(Max Frame) + (PFC Frame)• Headroom = DV * alpha• alpha is implementation dependent,
considering buffer chunk size

T1T3

Next Steps• Define PFC headroom measurement mechanism• Measurement capability advertisement• Measurement timestamp point• Measurement frame interaction• Measurement frame format• Calculation method with timestamp• Consideration of changes to 802.1Q

THANK YOU