Original text:
frames should therefore not be paced, to avoid delaying their delivery to the peer.  As an example of a well-known and publicly available implementation of a flow pacer, implementers are referred to the Fair Queue packet scheduler (fq qdisc) in Linux (3.11 onwards).

Reviewer comments:
Came from the discussion of
NAME : I realized after we did this PR yesterday that this does not I It reduces burstiness in general, which is still a good thing, but it doesn't change slow start increase. I'll send a PR or ask Vidhi if she wants to.
When an ACK arrives that acknowledges N packets (where N could be very large), QUIC will increase the congestion window by more than 1MSS per congestion window during Congestion Avoidance causing bursty send. This could also affect the slow start. if (congestionwindow < ssthresh): // Slow start. congestionwindow += URL else: // Congestion avoidance. congestionwindow += kMaxDatagramSize * URL / congestionwindow One could argue that QUIC ACKs every other packet, but there ACK packet could get lost or ACK for ACK packet could get lost. The draft is probably assuming packet pacing but not all implementations support that. I discussed this with Jana and he thinks that we should add something in pseudo code or some specific text in the draft.
PR added more specific text on avoiding bursts into the network, FYI.
Please note that PR addresses the issue of app-limited or pacing-limited cases to not increase cwnd when it is under utilized. My concern is cwnd being inflated due to an ACK frame containing ACKs for large number of packets.
NAME : The problem here is that we don't talk about burstiness that can occur when pacing is not in place, and a stretch-ack is received. PRR handles this, but we don't do PRR in the draft.
No we need to spend lots of time on Reno as everybody appears to be doing Cubic or BBR?
Cubic has this problem. BBR cannot be assumed.
Except Cubic opens the window based on time spent, not just ack received
I don't get the scenario. It seems that NAME is worried about receiving an ACK for a large number of packets. This is not a congestion window problem, because it happens even if the congestion windows size remains constant. The root cause is that the ACK of N packets now reduces the number of packet in flow by N. So this is strictly about pacing, not about congestion control.
NAME yes, and it can and still does burst because of stretch acks
Not just stretch Acks -- deliberate Ack compression by the network happens too. Very few alternative to pacing.
URL describes
Discussed in Cupertino. NAME to prepare a PR that hopefully can incorporate some ideas from RFC3465 to add safety features for non-pacing stacks.
But 3465 does not solve the burst problem. It limits the growth of the window, but does not do pacing of any kind.
I think we can change to something like this
NAME I think we need to do 2MSS clamping per ACK received and not per packet acked. I am thinking that we can remove OnPacketAckedCC and instead do OnAckReceivedCC. Pseudocode change something like this:
I think we can keep but just move cwnd update to as you propose.
Yup, that's an important missing fix, though it needs to be applied only when not pacing. Additionally, helps with burstiness in general. NAME : I would make a variable, and apply the min here when not pacing.
The proposed change has the effect of reducing the effective Window if ACKs are compressed. Say you have a window of 12 packets, acks are delayed, and a single ACK acknowledges 12 packets. Now you have a window of 2 packets, your transmission rate is 6 times slower than before, and it will take you 10 RTT to recover. Is that really what you want?
NAME This is what ABC (RFC 3465) does for TCP. ACK compression has side-effects, and the way to deal with it is, as you say, to pace packets out. This reduction in performance is only when the endpoint does not pace, and it is the only way to limit bursts in that case.
Also, ACK compression does not necessarily imply ACK pruning. I have traces in which 1 RTT worth of ACKs is compressed and delivered back to back. In that case, the proposed algorithm will do nothing: you will get 2 MSS added for each ACK, back to back within less than 1 ms.
FYI. FreeBSD implements rfc3465 (and enabled by default) URL Linux removed tcpabc implementation but saying it does in a different way: URL Considering FreeBSD doesn't have pacing by default, I think current proposed algorithm is not worse (or aggressive) than what tcp implementation does.
NAME I am working on the PR for pseudocode changes.
I agree that when not pacing, using an ABC limit is appropriate. FWIW Windows TCP was using a limit of 4 and not 2 for a long time and now uses 8 which is pretty close to IW10 burst in slow start phase.
For the WG: The proposed PR attempts to include a mechanism to limit CWND increase during slow start when not pacing. I'll note that by itself, it does not guarantee we follow the existing MUST: "Implementations MUST either use pacing or limit such bursts to the initial congestion window, which is recommended to be the minimum of 10 maxdatagramsize and max(2 maxdatagramsize, 14720)), where maxdatagramsize is the current maximum size of a datagram for the connection, not including UDP or IP overhead." I don't think we should add this to the pseudocode, because we say SHOULD pace packets and the pseudocode currently only covers SHOULDs, not MAYs. Adding all MAYs to the pseudocode is too complex. I don't have a concern about adding a reference to the existing RFC as a mechanism that MAY help limit bursts during slow start. I'd like a decision from the WG on whether to include this change and if so, some criteria on what should be included going forward.
I agree with the comment that burst from ACK compression or Stretch-ACKs result in bursts. I think the ways to change that is implement a maximum burst size or use pacing. The bursts do not appear a CC function to me.
Bringing conversation in from the list: With regards to , one of the things we found in testing was that it could (a) still allow an implementation to be vulnerable to too much bursting (the original problem), but more importantly (b) locked receivers into an ack pattern of acking every other packet. My main concern is that this causes problems if receivers want to experiment with and change acking strategies going forward in way that otherwise would be fine. One way to look at the problem described by this issue is that the existing text has a minor underspecification, in which it has the notion of a burst limit but says nothing about the frequency of such bursts. Two "bursts" of 10MSS that are sent within a couple nanoseconds are really one burst that's too big. Another way of fixing this is to remove even the specification of the 10MSS limit if we really don't want too much text about the non-pacing cases. Essentially, what this comes down to is: pace, or even if you don't pace, you need to not send bursts above some frequency; and if you don't set some pacing timers, you'll likely underutilize the link.
I'm going to put this out there: the answer is MUST pace. You might formalize this by saying the number of packets sent in any interval cannot exceed by more than the initial cwnd (i.e., the 10MSS). That allows for bursting, but requires a period of quiet either preceding or following any burst.
NAME Yes, it does largely reduce to "implementations MUST pace" (even if the pacing algorithm isn't very nuanced, in which case you'll perform poorly). The formalization of the rate, one option for which was specified in , I think is the heart of what should be done here.
There are two routes that we can take: Specify clearly what 'delay` refers to in the below statement, which is done in OR Change MUST pace to SHOULD and remove the 10 MSS burst size limit and provide general recommendation if an implementation doesn't pace. Something like:
I'm sympathetic to saying MUST pace, and I like the formatlization that NAME suggests. I've heard some pushback on saying MUST pace in the past, but it might be worth giving it a shot again. NAME NAME NAME : What do you think?
That said... I will note that NAME 's formalization allows for a burst at the end of an RTT. Consider for example when a single ACK is received that acknowledges an entire window. That would be legal by this definition. I think that's ok.
Discussed in ZRH. Proposed resolution is to define and REQUIRE pacing.
NAME I think requiring pacing makes good sense, with some formalization of what that means.
To clarify the status here, my understanding is that the proposal is to close this design issue with no action. The discussions have identified editorial improvements that will be addressed independently.
Let's continue the editorial discussion on .
Closing this issue and continuing discussion on
This needs to run through the consensus call procedure, so we'll keep it open for another week.
Let's get this in -- it addresses some burstiness anyway.LGTM

Revised text:
frames should therefore not be paced, to avoid delaying their delivery to the peer. <ins> Sending multiple packets into the network without any delay between them creates a packet burst that might cause short-term congestion and losses.  Implementations MUST either use pacing or limit such bursts to minimum of 10 * kMaxDatagramSize and max(2* kMaxDatagramSize, 14720)), the same as the recommended initial congestion window. </ins> As an example of a well-known and publicly available implementation of a flow pacer, implementers are referred to the Fair Queue packet scheduler (fq qdisc) in Linux (3.11 onwards).