Headroom Measurement
Protocol Design
Lily Lv (Huawei)
Fei Chen (Huawei)1

To-Do List
• Timestamp point clarification
 Need model/figure with labeled time points• Protocol design of request-response measurement• Managed objects
 The effort, implementation cost, and purpose of statistic gathering and retention requires careful consideration• DCBX：PFC Configuration TLV format design ----- more generic way???
 PFC configuration TLV defines Capability (round-trip, PTP-based) PFC informational TLV defines compensation value of PTP-based method

Conclusions: Ethertype for Qdt
 Reuse Qcz (CI) Ethertype 89-A2

 Timestamp accuracy Describe accuracy by number of pause quantas or number of maximum length frames, instead of number of nansseconds.
2

Timestamp Points Clarification

3

PFC Timestamp Points (Non-MACsec)
RX Queuet1
t4

TX Queue

PFC

PFC

t2
t3

MAC Control
MAC
PHY

MAC Control
MAC
PHY
Medium• PFC Headroom = t2-t1+ t4-t3• t1: RX queue is above threshold and invokes signal to PFC module.
• t2: TX queue receives signal from PFC module and stops transmission.
• t3: last packet is sent after TX queue is stopped• t4: last packet is received by RX queue

4

PFC Timestamp Points (Non-MACsec)
t1
t4

Measurement

Measurement

RX Queue

TX Queue

PFC

PFC

t2

TX QueueRX QueueRX Queue

TX Queuet3
MAC Control
MAC
PHY

MAC Control
MAC
PHY

MAC
PHY

MAC
PHY

MediumMedium• Different procedure• PFC invocation-> traffic stop vs. Measurement request-> measurement response• MAC control frame vs. MAC data frame• PFC pause frame takes the ‘quick path’ ------ > no data path delay, ‘quick path’ delay can be ignored• PFC pause frame waits at most 1 MAC data frame to be sent ------ > t2-t1 is variable, consider worst case
• After PFC is taken action, at most one more MAC data frame is sent ------ > t4-t3 is variable, consider worst case5

Figure N-3 Helps to Define Measurement Timestamp
Points (Non-MACsec)
RX Queue

TX Queue

t1
t4

PFC

PFC

t2
t3

MAC Control
MAC
PHY

MAC Control
MAC
PHY

802.1Q

MediumDelay Value = 2*(Cable Delay) + TXds1 + RXds2 + HDs2 + TXds2 + RXds1+ 2*(Max Frame) + (PFC Frame)6

Updated Figure N-3 (Non-MACsec)(PFC invocation
delay is ignored)

RX queue is above
threshold⓪

PFC M_CONTROL.request is
invoked, but a max length
frame just started transmission①

PFC frame begins
transmission after max
length fram completed②③

RXd𝑠2 BTs + Hd𝑠2 BTs

Station 2

Station 1

Last bit of PFC frame
passed to MAC service

Invocation delay BTs
16160 BTs (Maxframe size + IPG+
SFD/Preamble
672 BTs (PFC frame+ IPG+
SFD/Preamble
TXds1 BTs

Last bit of PFC frame
sent into the wire④
Last bit of PFC frame

RXds2 BTsHDs2 BTs(PFC reaction delay)
16160 BTs (Maxframe size + IPG+
SFD/Preamble
HDs2’ BTs
(data path delay)
TXds2 BTs

⑤ received from the wire
Last bit of PFC frame passed⑤’ from MAC service⑥⑦

PFC M_CONTROL.indicaton
generated Priority is paused,
but max length frame juststarted transmission
Last bit of maximum length
frame processed bytransmission selection

bit of frame passed to⑧ Last
MAC service
bit of frame sent into
⑨ Lastthe wire

Last bit from station 2
received from the wire⑩

Last bit from station 2
received and queued.

⑪

RXds1 BTs

7

PFC Timestamp Points in New Figure N-3 (Nont1⓪
MACsec)Station 2

Station 1

RX queue is above
thresholdInvocation delay BTs

PFC M_CONTROL.request is
invoked, but a max length
frame just started transmission①

RX Queuet1
t4

PFC frame begins
transmission after max
length fram completed

TX Queue

PFC

PFC

t2

③

Last bit of PFC frame
passed to MAC servicet3
MAC Control
MAC
PHY

②

672 BTs (PFC frame+ IPG+
SFD/Preamble
TXds1 BTs

Last bit of PFC frame
sent into the wire

MAC Control
MAC
PHY

④
Last bit of PFC frame

RXds2 BTsHDs2 BTs(PFC reaction delay)

Medium••••16160 BTs (Maxframe size + IPG+
SFD/Preamble

16160 BTs (Maxframe size + IPG+
SFD/Preamble

t1: RX queue is above threshold and invokes signal to PFC module.
t2: TX queue receives signal from PFC module and stops transmission.
t3: last packet is sent after TX queue is stopped
t4: last packet is received by RX queue
Last bit from station 2
received from the wire
Last bit from station 2
received and queued.

HDs2’ BTs
(data path delay)
TXds2 BTs

⑤ received from the wire
Last bit of PFC frame passed⑤’ from MAC service

t2 ⑥
t3 ⑦

PFC M_CONTROL.indicaton
generated Priority is paused,
but max length frame juststarted transmission
Last bit of maximum length
frame processed bytransmission selection

bit of frame passed to⑧ Last
MAC service
bit of frame sent into
⑨ Lastthe wire⑩⑪ t4

RXds1 BTs

8

PFC Headroom Calculation (Non-MACsec)
RX Queuet1
t4

TX Queue

PFC

PFC

t2
t3

MAC Control
MAC
PHY

MAC Control
MAC
PHY
Medium• PFC Headroom = t2-t1+ t4-t3• t1: RX queue is above threshold and invokes
signal to PFC module.
• t2: TX queue receives signal from PFC moduleand stops transmission.
• t3: last packet is sent after TX queue is stopped• t4: last packet is received by RX queue• PFC Headroom = t2-t1+ t4-t3 + 2*(Max Frame)• t1: RX queue is above threshold and invokes signal to
PFC module• t2: PFC M_CONTROL.indicaton generated. Priority ispaused, but max length frame just started transmission• t3: last bit of maximum length frame processed bytransmission selection• t4: last bit of frame received and queued
9Measurement Timestamp Points (Non-MACsec)
Station 1

RX queue is above
threshold⓪

t1

PFC M_CONTROL.request is
invoked, but a max length
frame just started transmission①

Measurement

Measurement
TX QueueRX QueueRX Queue

TX Queue

PFC frame begins
transmission after max
length fram completed

②
③ t1’

Last bit of PFC frame
passed to MAC service

t4’

t2’

t1’

Last bit of PFC frame
sent into the wire•••

16160 BTs (Maxframe size + IPG+
SFD/Preamble
672 BTs (PFC frame+ IPG+
SFD/Preamble④
Last bit of PFC frame

MAC
PHY

RXds2 BTsHDs2 BTs(PFC reaction delay)

Medium

•Invocation delay BTs

TXds1 BTs

t3’
MAC
PHY

Station 2

t1’: last bit of measurement req frame passed to MAC service• t1’ – PFC invocation delay – PFC frame = t1t2’: last bit of measurement req frame received from MAC service• t2’ + PFC reaction delay= t2t3’: last bit of measurement resp frame processed by transmission selection• t3’ = t3
t4: last bit of measurement resp frame received and queued• t4’ = t416160 BTs (Maxframe size + IPG+
SFD/Preamble
HDs2’ BTs
(data path delay)
TXds2 BTs

⑤ received from the wire
bit of PFC frame passed
t2’ ⑤’ Last
from MAC service

t2 ⑥
t3 ⑦
t3’

PFC M_CONTROL.indicaton
generated Priority is paused,
but max length frame juststarted transmission
Last bit of maximum length
frame processed bytransmission selection

bit of frame passed to⑧ Last
MAC service
bit of frame sent into
⑨ Lastthe wire

Last bit from station 2
received from the wire⑩

Last bit from station 2
received and queued.

⑪ t4
t4’

RXds1 BTs

10

PFC Headroom Calculation by Measurement Timestamp
Points (Non-MACsec)
t1
t4

Measurement

Measurement

RX Queue

TX Queue

PFC

PFC

t2

t3
MAC Control
MAC
PHY

TX QueueRX Queuet4’

MAC Control
MAC
PHY

RX Queue

TX Queue

t3’
t2’

t1’
MAC
PHY

MAC
PHY

MediumMedium••PFC Headroom = t2-t1+ t4-t3 + 2*(Max Frame)• t1: RX queue is above threshold and invokes signal to
PFC module• t2: PFC M_CONTROL.indicaton generated. Priority ispaused, but max length frame just started
transmission• t3: last bit of maximum length frame processed bytransmission selection• t4: last bit of frame received and queued

PFC Headroom = (t2’+ PFC reaction delay) –(t1’-PFC invocation delay –PFC frame) +t4’-t3’ + 2*(Max Frame)• t1’: last bit of measurement req frame passed to MAC service• t1’ – PFC invocation delay – PFC frame = t1
• t2’: last bit of measurement req frame received from MAC service• t2’ + PFC reaction delay= t2• t3’: last bit of measurement resp frame processed by transmission
selection• t3’ = t3
• t4: last bit of measurement resp frame received and queued• t4’ = t4
11

Implementation Example (Non-MACsec)
Measurement

Measurement

t1’’

t4'’
TX QueueRX QueueRX Queue

TX Queuet4’

Measurement

Measurement
TX QueueRX Queue

t2’’

t3’’

RX Queue

TX Queue

t3’
t2’

t1’
MAC
PHY

MAC
PHY
Medium

MAC
PHY

MAC
PHY
Medium•PFC Headroom = (t2’+ PFC reaction delay) –(t1’-PFC invocation delay-PFC frame) +t4’-t3’ + 2*(Max Frame)= (t2’’ – r_rx data path delay + PFC reaction time) – (t1’’ + l_tx data path delay – PFC invocation delay-PFC frame)+(t4’’ – l_rx data path delay )-(t3’’ + r_tx data path delay ) + 2*(Max Frame)• t1’’: last bit of req frame is generated by measurement module• t1’’ + l_tx data path delay = t1’• t2’’: last bit of req frame is received by measurement module• t2’’ – r_rx data path delay = t2’• t3’’: last bit of resp frame is generated by measurement module
Note： r_tx data path delay and l_rx data path
• t3’’ + r_tx data path delay = t3’
delay are not full data path delay.
• t4’: last bit of resp frame is received by measurement module• t4’’ – l_rx data path delay = t4’
12

PFC Timestamp Points (MACsec)
shim
TX Queueshim

RX Queuet1

TX Queue

PFC

PFC

t4

t2

RX Queuet3
MACsecMAC
PHY

MACsecMAC
PHY
Medium• PFC Headroom = t2-t1+ t4-t3• t1: RX queue is above threshold and invokes signal to PFC module.
• t2: TX queue receives signal from PFC module and stops transmission.
• t3: last packet is sent after TX queue is stopped• t4: last packet is received by RX queue

13

PFC Timestamp Points (MACsec)
shim
TX QueueRX Queuet1

TX Queue

PFC

PFC

t4

t2

Measurement

Measurementshim
RX Queue

TX QueueRX QueueRX Queue

TX Queuet3
MACsecMAC
PHY

MACsec
MAC
PHY

MAC
PHY

MAC
PHY

MediumMedium• Different procedure• PFC invocation-> traffic stop vs. Measurement request-> measurement response• MAC control frame vs. MAC data frame• PFC pause frame traverses on ‘quick path’ ------ > no data path delay• PFC pause frame waits at most 1 MAC data frame to be sent ------ > t2-t1 is variable• After PFC is taken action, at most one more MAC data frame is sent ------ > t4-t3 is variable
14

PFC Timestamp Points in New Figure N-3 (MACsec)
Station 2

Station 1

RX queue is above
threshold⓪
PFC M_CONTROL.request is invoked ①
Last bit of PFC frame passed
from shim layer, but max length
frame just start transmission

shim
TX Queuet1

TX Queue

PFC

PFC

t4

t2
t3

Shim delay BTs①’

PFC data frame begins
transmission after max length
frame completed
Last bit of PFC frame passed to
MAC service

16160 BTs (Maxframe size + IPG+
SFD/Preamble

RX QueueHDs1’ BTs (data path delay + MACsec delay)③
TXds1 BTs

Last bit of PFC frame sent into
the wire④
RXds2 BTs +HDs2’’ (data path delay+ MACsec delay)

MACsecMAC
PHY

MACsec
MAC
PHY

Invocation delay BTs

②

shim

RX Queuet1

Shim delay + HDs2 BTs(PFC reaction delay)
16160 BTs (Maxframe size + IPG+
SFD/Preamble

MediumHDs2’ BTs
(data pathdelay+MACsec delay)
TXds2 BTs
Last bit from station 2
received from the wire
Last bit from station 2
received and queued.

Last bit of PFC frame
received from the wire⑤
Last bit of PFC frame passed⑤’ to shim layer
PFC M_CONTROL.indicaton

t2 ⑥ generated Priority is paused,
but max length frame juststarted transmission

t3 ⑦ Last bit of maximum length
frame processed bytransmission selection

bit of frame passed to⑧ Last
MAC service
bit of frame sent into
⑨ Lastthe wire⑩⑪ t4

RXds1 BTs

15

PFC Headroom Calculation (MACsec)
shim
TX Queueshim

RX Queuet1

TX Queue

PFC

PFC

t4

t2

RX Queuet3
MACsecMAC
PHY

MACsecMAC
PHY
Medium• PFC Headroom = t2-t1+ t4-t3• t1: RX queue is above threshold and invokes
signal to PFC module.
• t2: TX queue receives signal from PFC moduleand stops transmission.
• t3: last packet is sent after TX queue is stopped• t4: last packet is received by RX queue• PFC Headroom = t2-t1+ t4-t3 + 2*(Max Frame)• t1: RX queue is above threshold and invokes signal to
PFC module• t2: PFC M_CONTROL.indicaton generated. Priority ispaused, but max length frame just started transmission• t3: last bit of maximum length frame processed bytransmission selection• t4: last bit of frame received and queued
16

Measurement Timestamp Points (Non-MACsec)
Station 1

RX queue is above
thresholdt1

Last bit of PFC frame passed
from shim layer, but max length
frame just start transmission①’ t1’⓪
PFC M_CONTROL.request is invoked ①
Measurement

Measurement

t2’

t1’
TX QueueRX QueueRX Queue

TX Queuet4’

PFC data frame begins
transmission after max length
frame completed
Last bit of PFC frame passed to
MAC service

Invocation delay BTs

②
③
④

MAC
PHY

Shim delay + HDs2 BTs(PFC reaction delay)••t1’: last bit of req frame is passed from measurement module• t1’ – PFC invocation delay – l_tx_shim layer delay = t1t2’: last bit of req frame is passed to measurement module• t2’ + r_rx_shim layer delay + PFC reaction delay = t2t3’: last bit of measurement resp frame processed by transmission selection• t3’ = t3t4’: last bit of measurement resp frame received and queued• t4’ = t416160 BTs (Maxframe size + IPG+
SFD/Preamble
HDs2’ BTs
(data pathdelay+MACsec delay)
TXds2 BTs
Last bit from station 2
received from the wire⑩

Last bit from station 2
received and queued.

⑪ t4
t4’

Last bit of PFC frame
received from the wire⑤
bit of PFC frame passed
t2’ ⑤’ Last
to shim layer

RXds2 BTs +HDs2’’ (data path delay+ MACsec delay)

Medium•16160 BTs (Maxframe size + IPG+
SFD/Preamble

TXds1 BTs
Last bit of PFC frame sent into
the wire•

Shim delay BTs

HDs1’ BTs (data path delay + MACsec delay)

t3’

MAC
PHY

Station 2

PFC M_CONTROL.indicaton

t2 ⑥ generated Priority is paused,
but max length frame juststarted transmission

t3 ⑦ Last bit of maximum length
frame processed byt3’transmission selection
bit of frame passed to⑧ Last
MAC service
bit of frame sent into
⑨ Lastthe wire

RXds1 BTs

17

Measurement Timestamp Points (MACsec)
shim
TX QueueRX Queuet1

TX Queue

PFC

PFC

t4

t2

RX Queue

MACsec
MAC
PHY

t2’

t1’
TX QueueRX Queuet3

MACsecMAC
PHY

Measurement

Measurementshim

RX Queue

TX Queue

t4’

t3’

MAC
PHY

MAC
PHY

MediumMedium•• PFC Headroom = (t2’ + r_rx_shim layer delay + PFC reaction delay ) –(t1’PFC Headroom = t2-t1+ t4-t3 + 2*(Max Frame)– PFC invocation delay – l_tx_shim layer delay) +t4’-t3’ + 2*(Max Frame)• t1: RX queue is above threshold and invokes signal to PFC• t1’: last bit of req frame is passed from measurement module
module• t1’ – PFC invocation delay – l_tx_shim layer delay = t1
• t2: PFC M_CONTROL.indicaton generated. Priority is paused,• t2’: last bit of req frame is passed to measurement module
but max length frame just started transmission• t2’ + r_rx_shim layer delay + PFC reaction delay = t2
• t3: last bit of maximum length frame processed by• t3’: last bit of measurement resp frame processed by transmission
transmission selection
selection• t4: last bit of frame received and queued• t3’ = t3
• t4’: last bit of measurement resp frame received and queued• t4’ = t418

Summary of PFC Timestamp Points

MACsecNon-MACsec

shim

RX Queuet1
t4

TX Queue

PFC

PFC
MAC Control
MAC
PHY

MAC Control
MAC
PHY
Medium

TX Queueshim

RX Queue

t2

t1

t3

t4

TX Queue

PFC

PFC

t2

RX Queuet3
MACsecMAC
PHY

MACsecMAC
PHY
Medium• PFC Headroom = t2-t1+ t4-t3 + 2*(Max Frame)• t1: RX queue is above threshold and invokes signal to PFC module• t2: PFC M_CONTROL.indicaton generated. Priority is paused, but max
length frame just started transmission• t3: last bit of maximum length frame processed by transmission selection• t4: last bit of frame received and queued
19

Summary of Measurement Timestamp Points

MACsecNon-MACsec

Measurement

Measurement
TX QueueRX QueueRX Queue

TX Queuet4’

TX Queuet1’

RX Queuet2’

RX Queue

TX Queue

t4’

t3’

t3’

t2’

t1’
MAC
PHY

MAC
PHY

MAC
PHY

MAC
PHY
MediumMedium•Measurement

Measurement•
PFC Headroom = (t2’+ PFC reaction delay) –(t1’-PFC invocation
delay-PFC frame) +t4’-t3’ + 2*(Max Frame)• t1’: last bit of measurement req frame passed to MAC
service• t2’: last bit of measurement req frame received and queued• t3’: last bit of measurement resp frame processed bytransmission selection• t4: last bit of measurement resp frame received and queued

PFC Headroom = (t2’ + r_rx_shim layer delay + PFC reaction delay ) –(t1’ – PFC invocation delay – l_tx_shim layer delay) +t4’-t3’ + 2*(MaxFrame)• t1’: req frame is passed from measurement module• t2’: req frame is passed to measurement module• t3’: last bit of measurement resp frame processed bytransmission selection• t4’: last bit of measurement resp frame received and queued

Should the timestamp points（t1’, t2’） converged?

20

Thanks

21

Done: PFC Configuration TLV format design• Proposal : PFC configuration TLV only includes ‘capability’

Each bit indicates one
capability.

Define priority of the 2 methods.

If non-PTP and PTP-based are supported on both sides, each node choose its own preference.

 ‘PTP comp’ for PTP-based measurement passes to peer separately.
Define a new informational TLV - PFC informational TLV

Compensation value for
PTP-based measurement

DCBX informational attributes:“Informational attributes areexchanged via LLDP without
any participation in a DCBX
state machine.”4Timestamp Point Clarification (1/2)

Without MACsec

PFC reaction time

Roundtrip delay

t1

Delay Value = 2*(Cable Delay) + TXds1 + RXds2 + HDs2 + TXds2 + RXds1

t2+ 2*(Max Frame) + (PFC Frame)

t4

t2’

t3’ t3Higher layer delay

t1’

Interface delay

t1: last bit of measurement request message passed to MAC service
t4: last bit of measurement response message passed from MAC service
t2: last bit of measurement request message passed from MAC service
t3: last bit of measurement response message passed to MAC service
Roundtrip delay = t4 – (t1’ – (MAC control processing time) )– (t3 – (t2 + (MAC control processing time))+ (PFC reaction time)≈ t4 – t1 – (t3 – t2)
Modified model based on 802.1Q Figure N-2—Delay modelTimestamp Point Clarification (2/2)
With MACsec

Roundtrip delay
Delay Value = 2*(Cable Delay) + TXds1 + RXds2 + HDs2 + TXds2 + RXds1+ 2*(Max Frame) + (PFC Frame)

t2

t4

t3

Higherlayer delay

t1

Interface delay

t1: last bit of measurement request message passed to MAC service
t4: last bit of measurement response message passed from MAC service
t2: last bit of measurement request message passed from MAC service
t3: last bit of measurement response message passed to MAC service
Roundtrip delay = t4 – (t1 – (shim processing time) )– (t3 – (t2 + (shim processing time))+ (PFC reaction time)≈ t4 – t1 – (t3 – t2)
Modified model based on 802.1Q Figure N-2—Delay modelTimestamp Accuracy• Local clock frequency drift analysis
Assume 5ppm oscillator, fiber cable 100Gbps and 10km link distance：(t4-t1) is no more than 200us : 100us link delay plus internal processing delay
1ns time offset in 200us, can be ignored.

• Captured timestamp point analysis
Expected timestamp point:

Implementation example:
t1 = t1’ + ePP delay
t4 = t4’ – iPP delay
t1 = t1’ + ePP delay
t4 = t4’ – iPP delay

t1: last bit of measurement request message passed to MAC service
t4: last bit of measurement request message passed from MAC service
t2: last bit of measurement request message passed from MAC service
t3: last bit of measurement request message passed to MAC service

PFC initiator

PFC receiver