Change:
I believe this also fixes the remaining portion of
When an ACK arrives that acknowledges N packets (where N could be very large), QUIC will increase the congestion window by more than 1MSS per congestion window during Congestion Avoidance causing bursty send. This could also affect the slow start. if (congestionwindow < ssthresh): // Slow start. congestionwindow += URL else: // Congestion avoidance. congestionwindow += kMaxDatagramSize * URL / congestionwindow One could argue that QUIC ACKs every other packet, but there ACK packet could get lost or ACK for ACK packet could get lost. The draft is probably assuming packet pacing but not all implementations support that. I discussed this with Jana and he thinks that we should add something in pseudo code or some specific text in the draft.
PR added more specific text on avoiding bursts into the network, FYI.
Please note that PR addresses the issue of app-limited or pacing-limited cases to not increase cwnd when it is under utilized. My concern is cwnd being inflated due to an ACK frame containing ACKs for large number of packets.
NAME : The problem here is that we don't talk about burstiness that can occur when pacing is not in place, and a stretch-ack is received. PRR handles this, but we don't do PRR in the draft.
No we need to spend lots of time on Reno as everybody appears to be doing Cubic or BBR?
Cubic has this problem. BBR cannot be assumed.
Except Cubic opens the window based on time spent, not just ack received
I don't get the scenario. It seems that NAME is worried about receiving an ACK for a large number of packets. This is not a congestion window problem, because it happens even if the congestion windows size remains constant. The root cause is that the ACK of N packets now reduces the number of packet in flow by N. So this is strictly about pacing, not about congestion control.
NAME yes, and it can and still does burst because of stretch acks
Not just stretch Acks -- deliberate Ack compression by the network happens too. Very few alternative to pacing.
URL describes
Discussed in Cupertino. NAME to prepare a PR that hopefully can incorporate some ideas from RFC3465 to add safety features for non-pacing stacks.
But 3465 does not solve the burst problem. It limits the growth of the window, but does not do pacing of any kind.
I think we can change to something like this
NAME I think we need to do 2MSS clamping per ACK received and not per packet acked. I am thinking that we can remove OnPacketAckedCC and instead do OnAckReceivedCC. Pseudocode change something like this:
I think we can keep but just move cwnd update to as you propose.
Yup, that's an important missing fix, though it needs to be applied only when not pacing. Additionally, helps with burstiness in general. NAME : I would make a variable, and apply the min here when not pacing.
The proposed change has the effect of reducing the effective Window if ACKs are compressed. Say you have a window of 12 packets, acks are delayed, and a single ACK acknowledges 12 packets. Now you have a window of 2 packets, your transmission rate is 6 times slower than before, and it will take you 10 RTT to recover. Is that really what you want?
NAME This is what ABC (RFC 3465) does for TCP. ACK compression has side-effects, and the way to deal with it is, as you say, to pace packets out. This reduction in performance is only when the endpoint does not pace, and it is the only way to limit bursts in that case.
Also, ACK compression does not necessarily imply ACK pruning. I have traces in which 1 RTT worth of ACKs is compressed and delivered back to back. In that case, the proposed algorithm will do nothing: you will get 2 MSS added for each ACK, back to back within less than 1 ms.
FYI. FreeBSD implements rfc3465 (and enabled by default) URL Linux removed tcpabc implementation but saying it does in a different way: URL Considering FreeBSD doesn't have pacing by default, I think current proposed algorithm is not worse (or aggressive) than what tcp implementation does.
NAME I am working on the PR for pseudocode changes.
I agree that when not pacing, using an ABC limit is appropriate. FWIW Windows TCP was using a limit of 4 and not 2 for a long time and now uses 8 which is pretty close to IW10 burst in slow start phase.
For the WG: The proposed PR attempts to include a mechanism to limit CWND increase during slow start when not pacing. I'll note that by itself, it does not guarantee we follow the existing MUST: "Implementations MUST either use pacing or limit such bursts to the initial congestion window, which is recommended to be the minimum of 10 maxdatagramsize and max(2 maxdatagramsize, 14720)), where maxdatagramsize is the current maximum size of a datagram for the connection, not including UDP or IP overhead." I don't think we should add this to the pseudocode, because we say SHOULD pace packets and the pseudocode currently only covers SHOULDs, not MAYs. Adding all MAYs to the pseudocode is too complex. I don't have a concern about adding a reference to the existing RFC as a mechanism that MAY help limit bursts during slow start. I'd like a decision from the WG on whether to include this change and if so, some criteria on what should be included going forward.
I agree with the comment that burst from ACK compression or Stretch-ACKs result in bursts. I think the ways to change that is implement a maximum burst size or use pacing. The bursts do not appear a CC function to me.
Bringing conversation in from the list: With regards to , one of the things we found in testing was that it could (a) still allow an implementation to be vulnerable to too much bursting (the original problem), but more importantly (b) locked receivers into an ack pattern of acking every other packet. My main concern is that this causes problems if receivers want to experiment with and change acking strategies going forward in way that otherwise would be fine. One way to look at the problem described by this issue is that the existing text has a minor underspecification, in which it has the notion of a burst limit but says nothing about the frequency of such bursts. Two "bursts" of 10MSS that are sent within a couple nanoseconds are really one burst that's too big. Another way of fixing this is to remove even the specification of the 10MSS limit if we really don't want too much text about the non-pacing cases. Essentially, what this comes down to is: pace, or even if you don't pace, you need to not send bursts above some frequency; and if you don't set some pacing timers, you'll likely underutilize the link.
I'm going to put this out there: the answer is MUST pace. You might formalize this by saying the number of packets sent in any interval cannot exceed by more than the initial cwnd (i.e., the 10MSS). That allows for bursting, but requires a period of quiet either preceding or following any burst.
NAME Yes, it does largely reduce to "implementations MUST pace" (even if the pacing algorithm isn't very nuanced, in which case you'll perform poorly). The formalization of the rate, one option for which was specified in , I think is the heart of what should be done here.
There are two routes that we can take: Specify clearly what 'delay` refers to in the below statement, which is done in OR Change MUST pace to SHOULD and remove the 10 MSS burst size limit and provide general recommendation if an implementation doesn't pace. Something like:
I'm sympathetic to saying MUST pace, and I like the formatlization that NAME suggests. I've heard some pushback on saying MUST pace in the past, but it might be worth giving it a shot again. NAME NAME NAME : What do you think?
That said... I will note that NAME 's formalization allows for a burst at the end of an RTT. Consider for example when a single ACK is received that acknowledges an entire window. That would be legal by this definition. I think that's ok.
Discussed in ZRH. Proposed resolution is to define and REQUIRE pacing.
NAME I think requiring pacing makes good sense, with some formalization of what that means.
To clarify the status here, my understanding is that the proposal is to close this design issue with no action. The discussions have identified editorial improvements that will be addressed independently.
Let's continue the editorial discussion on .
Closing this issue and continuing discussion on
This needs to run through the consensus call procedure, so we'll keep it open for another week.
"Bursting more than an intial window’s worth of data into the network might cause short-term congestion and losses." • Is that really "more than...", I think the evidence was that a larger IW (as in QUIC) can lead to congestion. • We could explain burst size and scenarios, or could simply say something like: "Bursting an initial window’s worth of data into the network can cause short-term congestion and losses for some path conditions." (This also fixes /intial/initial)?
I think we should address this in PR
Linux pacing has an initial burst of 10 packets, followed by pacing, so this is intended to allow for that style of sending. In which case, I think "more than" is the right term? I think the evidence you have in mind is some data presented early on that pacing + IW32 had lower losses than no pacing and IW10? That gap largely is driven by the large initial burst out of quiescence without pacing, which would be mitigated by sending a 10 packet burst and pacing the rest(or doing some form of slow start after idle).
Ah, my reading of the text led me in a different direction - I think we can now talk. ( I know Google shared data on IW10 and also later on other IW proposals. I also think I have seen other analysis that suggested that a larger IW can have an impact - IW10, or larger - can have collateral damage to flows sharing a bottleneck. In this case, because there is no ACK-clocking. ) Could I propose describing limiting the max burst size and enabling pacing for any remaining packets sent above this using the remaining (cwnd-maxburst). My reading is that the current text sets the maxburst size to IW. That is certainly one choice, and could be an upper limit. I am less clear that IW (the startup cwnd) should always be the same as the maximum burst size. Some SCTP implementation uses 4 packets, and I suspect that there could be cases where someone proposed a larger IW, but retained a max_burst that is lower than the increased IW.
I think making IW the max burst size, rather than the recommended burst size is sensible. I've certainly seen real-world traces where bursting 4 packets would not have caused loss, but 10 did. Text suggestions on the PR very welcome.
NAME I think you might be talking about a general max burst size, where the recovery draft (and NAME is trying to talk about an "initial" window's size -- the size of a burst after quiescence. Is that right? I think we need to do some wordsmiting, but I want to make sure that we're talking about the same thing first.
Is there such a thing. I'm wondering: if you use pacing, you still want to cluster your transmissions so you don't get too frequent timer interrupts unless the network adapter is capable of pacing. So how do you decide the granularity of bursts when pacing?
That's a different thing - pacing shouldn't mean you can't have (smallish) bursts.
In response to Jana: I think it would be a good step to first make sure we understand the same by IW - which is initial cwnd, and somewhat dependent on RTT - and a function of the capacity; and maxburst; which isn't related to RTT, but could be related to when it is helpful to "pace". As a starting point, RFC4960 recommends maxburst as 4. Would we agree for instance, largest(max_burst) <= IW?
The current text(before my outstanding PR) does not mention out of quiescence anymore. However, upon re-reading the text, I think it could be clearer, so I updated it in PR .
Gorry: Yes, I agree that micro-bursts ought to be smaller than IW. That's sensible, I think, but we really don't have a best practice for micro-bursts that we can point to (I just remembered that I had written a with mallman 14 years ago on trying to specify micro-burst behavior!). Yes, RFC 4960 uses a simple micro-burst of 4 segments, but that was based on RFC 3390, which used to limit the initial window to at most 4 segments. That's changed now, but I'd be wary of reflecting that change onto micro-bursts, because the analysis is quite different. So while I agree that largest(maxburst) <= IW, I think talking about maxburst adds something new to what the draft is currently talking about. The draft is simply using IW for both connection start-up and noting that as a known upper bound for bursts. That is not the same as specifying maxburst however. I think we ought to say something for senders coming out of an app limited period. (Perhaps we should say that bursting SHOULD be limited in the absence of pacing and add a note to various past maxburst limits without specifying a new one?) I'm not keen to specify a max_burst here (but I'm not sure if that's a tenable position).
I'm not advocating a max burst size of 4 (although I do believe this is safe, other sizes larger may also be safe), however my thought was that the draft infers IW as a safe size to burst - I don't know that this is always true - especially if we look to the future use of a larger IW. You could (I think you do) argue that IW is an upper bound to this safe burst size without pacing. I will try just a little more: • I think we should limit the cwnd when coming out of an app-limited period. • I think we should limit a burst <= some size or that pacing is needed, or if not then it MUST (!) reduce to a safe IW. Does that help?
changed the discussion in recovery from idle behavior and app limited periods to periods of under-utilized cwnd. This is great, but a precise definition of when the cwnd goes under-utilized is still missing.
Defining "under-utilised cwnd", seems conceptually the right thing, but we struggled with that term and the term "fully utilized the congestion window" when writing RFC7661, because ACKs can arrive in bursts and apps can do their work in bursts: Does that mean it's more than some threshold for some period time?
NAME : That is a question. Maybe we can get away from that by saying that a sender only increases its cwnd if it knows that it was fully utilized since the last cwnd change? Meaning that at some point since the last cwnd change, inflight >= cwnd happened at least once. I have a separate question, one that was being discussed in , and I think is still left open. If we need the entire cwnd to be used before increasing it, we lose out on 1 RTT of growth during slow start. Both Windows and Linux TCP compensate for this by doing something like increase it when > 1/2 cwnd is used... we need to write something here that's at least as performant, and not just "when inflight >= cwnd".
NAME thoughts?
This seems to be digging in the correct place. I think we would agree the aim of some form of cwnd-validation is to stop cwnd growth when there is no growth of inflight data by the upper layer. And I think we'd agree tussle comes when the CC is trying to expand the cwnd - because the upper layer has more data to send, but that doesn't mean cwnd was necessarily fully utilised last RTT. When would in-flight > cwnd ?
NAME Good point about the in_flight > 1/2 CWND check in Linux. In BBR, that only applies to the first doubling of IW, and after that it's fully bandwidth based, FWIW. We could add a similar constraint that this rule can be applied when the window was not fully used at the time of sending, but was at the time of acknowledgement as long as the window is < 2*IW? One clear goal of mine is I don't want the handshake alone increasing the congestion window(assuming IW10 and a moderately sized handshake).
Putting this discussion into a concrete algorithm: -Keep an in-flight high watermark, InFlightHigh, that's reset on each CWND change. -When considering growing the CWND, only let it go as large as InFlightHigh2. -Justification for the "2": it allows base-2 exponential growth, which is the fastest growth we ever want. EDIT: expire on CWND change might not work. I think InFlightHigh expiry is the hard part here.
Possible expiration events: persistent congestion, ssthresh changes, app idle.
Discussed in London, we'd like to avoid specifying this in pseudocode and the exact math, as it may be implementation dependent, but instead focus on improving the existing text.
Currently the recovery draft 19 requires the sender to either reducing the cwnd to initial cwnd OR pace the send after idle. Idle is defined as "it ceases to send data and has no bytes in flight" . This will mean that just a very small delay in the sending application thread will cause QUIC to reset its cwnd when not pacing. IMO we need to define an interval for this which could be a function of kGranularity or as PTO. FWIW in TCP the idle period for cwnd restart is defined as one retransmission timeout (reference URL Section 4.1).
However, this still has the problem that if I don't pace and I have a window of 100 packets, I can burst 100 packets into the network. This seems suboptimal from both the sender and the network's perspective. Now that I think about it, I think the concept of limiting the size of a burst is the key here, not so much idleness, and I think defining idle more accurately is both not necessary and likely to be difficult. Ideally you would send no more than max(10, BW * kTimerGranularity) packets into the network at once.
NAME pacing or limiting burst is the right idea. TCP RFC limits the burst to IW but it says do that after an idle period of an RTO. Presumably the authors and community considered that period to be small enough that network capacity / conditions hadn't changed drastically. Implementation wise how would we do this without a timer? If not pacing, when would one decide to cut cwnd back to restart cwnd (say IW)?
Your interpretation of RFC5661 looks correct and is different from what's in the QUIC recovery doc. But the implications are that it's ok to send an unlimited sized burst into the network as long as you only drained the pipe very recently. I think that recommendation is a poor one unless the window sizes are fairly small(in which case they'd be close to IW10 anyway). The sender is bursting packets into the network, which previously did not cause loss because they were ack-clocked, but one or more buffers on the network doesn't have enough space for the full CWND. It seems likely this could cause losses for other flows sharing the link as well. This could happen frequently in HTTP applications such as DASH video playback. NewCWV is already referenced for decaying CWND after idle. I could change the text to say nothing about reducing CWND to IW upon full idle and instead say you shouldn't burst more than IW into the network at once ever, and senders may use pacing, ACK-clocking, and/or timers to accomplish that? URL may also be relevant to this topic?
I guess the bigger question is what are we attempting to document in this draft. Is it whats in the TCP standard RFCs OR what most current TCP implementations do OR what the Linux kernel implementation does? By documenting NewReno instead of CUBIC we've chosen to not document the state-of-the-art and erred on the side of simplicity. In the spirit of that I think handling idle period and app limited states are optimizations for better network citizenship on cetrtain type of networks and not a hard requirement. For this particular issue I'd like to point out that the Windows TCP implementation has worked fine without handling the idle period (not capping cwnd burst and neither pacing). The idle cwnd restart knob in Windows is off by defualt and I believe Google TCP turns it off on Linux as well. Yes RFC 7661 is relevant and I think I cited it on the application limited case handling issue. I think there is a fine balance between managing burst size and taking advantage of TSO like optimizations. For example in WAN and DC bulk flows it might better to burst higher than IW with TSO for CPU efficiency than to pace and on the Internet pacing might be better. But I like your direction on this since it sidesteps what is meant by "idle". How about framing it as a suggestion and not a requirement. Something along the lines of: Bursts of more than IW into the network at once may cause losses. Implemementations should consider using pacing, ACK-clocking, and/or timers to minimize bursts. Paging NAME to this issue as well.
My goal is to document the best practices and reference existing RFCs whenever possible. And yes, my understanding is that everyone disables slow start after idle, so I'd like to come up with something I think is more likely to be deployed. When QUIC was started, Cubic wasn't an RFC, which is the other reason we documented Reno instead of Cubic, FYI. I'll write up a PR based on your suggestion and hopefully we can get some text we all think is acceptable.
RFC7661 is already cited in the section, and it was added there to defer to it so we didn't have to delve into details in this doc. I think what we're really discussing here is whether we need the MUST in the sentence "When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window". I'm in agreement with what I'm sensing here -- that we don't need a requirement, but we can discuss the implications of various decisions here and current best practices. Higher-level note, yeah, we always wanted to document current best practices, rather than what was in TCP or even linux TCP. I think most things came from linux TCP because that was what we were familiar with. FWIW, agreed that Cubic wasn't an RFC at the time, but it was close to becoming one, and we did discuss it. We decided that it was too complex and adds a ton more complexity to this document. We wanted to have something simple but good enough that implementations could have as a starting point, and then move on to more esoteric algorithms as deployments needed. (I'll disagree that Cubic is the best known CC at the moment -- unless you're talking about DC environments, it's not.)
Thanks for the discussion, I updated based on this, though I'm a bit concerned that SHOULD is too weak in regards to bursts into the network. Then again, if that's what Windows is doing, maybe we have to accept the reality of current deployments?
Given that I think that the general guidance for QUIC is "use pacing", I think a "MUST reset to IW if not paced" is actually the right thing to do to support that pacing gets implemented with QUIC from the start.
NAME pacing may be implemented and controlled outside of the transport. I have two new concerns about text around bursting packets on network: Many CDNs and edge deployments will tweak IW to be a higher value. In Windows TCP we allow config of up to 64 MSS. So if the network admin knows what they are doing and can swallow bursts then we shouldn't discourage the practice. UDP segmentation offload is reality now and for high throughput flows using hardware offload to reduce CPU will mean larger bursts trading off against pacing. We shouldn't preclude this either.
Discussed in Montreal; people to review the PR.
Discussed in Cupertino, NAME to review and determine whether that addresses (or partially addresses) this issue.
NAME NAME This issue was supposed to be but it was waiting for feedback. It's been long enough -- can we close this issue?

Discussion:
6.9. <del> A congestion window that is under-utilized SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </del> <ins> When bytes in flight is smaller than the congestion window and sending is not pacing limited, the congestion window is under- utilized.  When this occurs, the congestion window SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </ins> A sender MAY use the pipeACK method described in section 4.3 of RFC7661 to determine if the congestion window is sufficiently

sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay. <del> Bursting more than an initial window's worth of data into the network might cause short-term congestion and losses.  Implemementations SHOULD either use pacing or reduce their congestion window to limit such bursts. </del> <ins> Sending multiple packets into the network without any delay between them creates a packet burst that might cause short-term congestion and losses.  Implementations SHOULD either use pacing or reduce their congestion window to limit such bursts to minimum of 10 * kMaxDatagramSize and max(2* kMaxDatagramSize, 14720)), the same as the recommended initial congestion window. </ins> A sender MAY implement alternate mechanisms to update its congestion window after periods of under-utilization, such as those proposed for