Change:
Combines the section on sending after idle and application limited because they share a lot in common. Praveen, I think this is better, but tell me if you'd like to add/change something else? and
I'll take this in, and open an issue for defining "under-utilized cwnd"
changed the discussion in recovery from idle behavior and app limited periods to periods of under-utilized cwnd. This is great, but a precise definition of when the cwnd goes under-utilized is still missing.
Defining "under-utilised cwnd", seems conceptually the right thing, but we struggled with that term and the term "fully utilized the congestion window" when writing RFC7661, because ACKs can arrive in bursts and apps can do their work in bursts: Does that mean it's more than some threshold for some period time?
NAME : That is a question. Maybe we can get away from that by saying that a sender only increases its cwnd if it knows that it was fully utilized since the last cwnd change? Meaning that at some point since the last cwnd change, inflight >= cwnd happened at least once. I have a separate question, one that was being discussed in , and I think is still left open. If we need the entire cwnd to be used before increasing it, we lose out on 1 RTT of growth during slow start. Both Windows and Linux TCP compensate for this by doing something like increase it when > 1/2 cwnd is used... we need to write something here that's at least as performant, and not just "when inflight >= cwnd".
NAME thoughts?
This seems to be digging in the correct place. I think we would agree the aim of some form of cwnd-validation is to stop cwnd growth when there is no growth of inflight data by the upper layer. And I think we'd agree tussle comes when the CC is trying to expand the cwnd - because the upper layer has more data to send, but that doesn't mean cwnd was necessarily fully utilised last RTT. When would in-flight > cwnd ?
NAME Good point about the in_flight > 1/2 CWND check in Linux. In BBR, that only applies to the first doubling of IW, and after that it's fully bandwidth based, FWIW. We could add a similar constraint that this rule can be applied when the window was not fully used at the time of sending, but was at the time of acknowledgement as long as the window is < 2*IW? One clear goal of mine is I don't want the handshake alone increasing the congestion window(assuming IW10 and a moderately sized handshake).
Putting this discussion into a concrete algorithm: -Keep an in-flight high watermark, InFlightHigh, that's reset on each CWND change. -When considering growing the CWND, only let it go as large as InFlightHigh2. -Justification for the "2": it allows base-2 exponential growth, which is the fastest growth we ever want. EDIT: expire on CWND change might not work. I think InFlightHigh expiry is the hard part here.
Possible expiration events: persistent congestion, ssthresh changes, app idle.
Discussed in London, we'd like to avoid specifying this in pseudocode and the exact math, as it may be implementation dependent, but instead focus on improving the existing text.
Currently the recovery draft 19 requires the sender to either reducing the cwnd to initial cwnd OR pace the send after idle. Idle is defined as "it ceases to send data and has no bytes in flight" . This will mean that just a very small delay in the sending application thread will cause QUIC to reset its cwnd when not pacing. IMO we need to define an interval for this which could be a function of kGranularity or as PTO. FWIW in TCP the idle period for cwnd restart is defined as one retransmission timeout (reference URL Section 4.1).
However, this still has the problem that if I don't pace and I have a window of 100 packets, I can burst 100 packets into the network. This seems suboptimal from both the sender and the network's perspective. Now that I think about it, I think the concept of limiting the size of a burst is the key here, not so much idleness, and I think defining idle more accurately is both not necessary and likely to be difficult. Ideally you would send no more than max(10, BW * kTimerGranularity) packets into the network at once.
NAME pacing or limiting burst is the right idea. TCP RFC limits the burst to IW but it says do that after an idle period of an RTO. Presumably the authors and community considered that period to be small enough that network capacity / conditions hadn't changed drastically. Implementation wise how would we do this without a timer? If not pacing, when would one decide to cut cwnd back to restart cwnd (say IW)?
Your interpretation of RFC5661 looks correct and is different from what's in the QUIC recovery doc. But the implications are that it's ok to send an unlimited sized burst into the network as long as you only drained the pipe very recently. I think that recommendation is a poor one unless the window sizes are fairly small(in which case they'd be close to IW10 anyway). The sender is bursting packets into the network, which previously did not cause loss because they were ack-clocked, but one or more buffers on the network doesn't have enough space for the full CWND. It seems likely this could cause losses for other flows sharing the link as well. This could happen frequently in HTTP applications such as DASH video playback. NewCWV is already referenced for decaying CWND after idle. I could change the text to say nothing about reducing CWND to IW upon full idle and instead say you shouldn't burst more than IW into the network at once ever, and senders may use pacing, ACK-clocking, and/or timers to accomplish that? URL may also be relevant to this topic?
I guess the bigger question is what are we attempting to document in this draft. Is it whats in the TCP standard RFCs OR what most current TCP implementations do OR what the Linux kernel implementation does? By documenting NewReno instead of CUBIC we've chosen to not document the state-of-the-art and erred on the side of simplicity. In the spirit of that I think handling idle period and app limited states are optimizations for better network citizenship on cetrtain type of networks and not a hard requirement. For this particular issue I'd like to point out that the Windows TCP implementation has worked fine without handling the idle period (not capping cwnd burst and neither pacing). The idle cwnd restart knob in Windows is off by defualt and I believe Google TCP turns it off on Linux as well. Yes RFC 7661 is relevant and I think I cited it on the application limited case handling issue. I think there is a fine balance between managing burst size and taking advantage of TSO like optimizations. For example in WAN and DC bulk flows it might better to burst higher than IW with TSO for CPU efficiency than to pace and on the Internet pacing might be better. But I like your direction on this since it sidesteps what is meant by "idle". How about framing it as a suggestion and not a requirement. Something along the lines of: Bursts of more than IW into the network at once may cause losses. Implemementations should consider using pacing, ACK-clocking, and/or timers to minimize bursts. Paging NAME to this issue as well.
My goal is to document the best practices and reference existing RFCs whenever possible. And yes, my understanding is that everyone disables slow start after idle, so I'd like to come up with something I think is more likely to be deployed. When QUIC was started, Cubic wasn't an RFC, which is the other reason we documented Reno instead of Cubic, FYI. I'll write up a PR based on your suggestion and hopefully we can get some text we all think is acceptable.
RFC7661 is already cited in the section, and it was added there to defer to it so we didn't have to delve into details in this doc. I think what we're really discussing here is whether we need the MUST in the sentence "When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window". I'm in agreement with what I'm sensing here -- that we don't need a requirement, but we can discuss the implications of various decisions here and current best practices. Higher-level note, yeah, we always wanted to document current best practices, rather than what was in TCP or even linux TCP. I think most things came from linux TCP because that was what we were familiar with. FWIW, agreed that Cubic wasn't an RFC at the time, but it was close to becoming one, and we did discuss it. We decided that it was too complex and adds a ton more complexity to this document. We wanted to have something simple but good enough that implementations could have as a starting point, and then move on to more esoteric algorithms as deployments needed. (I'll disagree that Cubic is the best known CC at the moment -- unless you're talking about DC environments, it's not.)
Thanks for the discussion, I updated based on this, though I'm a bit concerned that SHOULD is too weak in regards to bursts into the network. Then again, if that's what Windows is doing, maybe we have to accept the reality of current deployments?
Given that I think that the general guidance for QUIC is "use pacing", I think a "MUST reset to IW if not paced" is actually the right thing to do to support that pacing gets implemented with QUIC from the start.
NAME pacing may be implemented and controlled outside of the transport. I have two new concerns about text around bursting packets on network: Many CDNs and edge deployments will tweak IW to be a higher value. In Windows TCP we allow config of up to 64 MSS. So if the network admin knows what they are doing and can swallow bursts then we shouldn't discourage the practice. UDP segmentation offload is reality now and for high throughput flows using hardware offload to reduce CPU will mean larger bursts trading off against pacing. We shouldn't preclude this either.
Discussed in Montreal; people to review the PR.
Discussed in Cupertino, NAME to review and determine whether that addresses (or partially addresses) this issue.
NAME NAME This issue was supposed to be but it was waiting for feedback. It's been long enough -- can we close this issue?
The text in recovery draft 19 says "The congestion window should not be increased in slow start or congestion avoidance when it is not fully utilized". This is a deviation from how congestion control works in deployed TCP implementation. We need a crisp definition of what is meant by "full utilization". I also have concern that this will make QUICs performance for bulk transfers poorer than TCP because application I/O patterns and scheduling of the sending thread will affect cwnd growth. After making the definition crisp and consensus that we need to include such a provision, the pseudo code will need to be updated to handle this as well.
NAME You're right, this needs to be made crisp and we need to add pseudo-code to handle this. Do you have a strawman proposal? How does Windows TCP do currently to handle growth during app limited periods? The naive approach doesn't work, since it prevents growth in the second RTT during slow start. I don't remember anymore what Linux TCP does, but I can look into that.
FWIW, my read of the Linux code tells me that at ack receipt time, the sender considers itself app-limited if in slow-start and cwnd > 2 x inflight, or if in cong-avoid and cwnd > inflight.
Jana, your reading matches my memory, but I thought there was an extra "fq-pacing quantum" added to inflight in the second case?
So there is no standard mechanism defined for this in any of the TCP RFCs especially the ones describing modern congestion control like CUBIC and DCTCP. Instead of providing a particular algorithm I propose we leave the handling of this to the implementations? Maybe we can refer implementers to URL for possible ideas.
I’m not sure, but I guess RFC4960 & SCTP implementation may be useful in this case.
I found this in RFC4960(URL): "When cwnd is less than or equal to ssthresh, an SCTP endpoint MUST use the slow-start algorithm to increase cwnd only if the current congestion window is being fully utilized, an incoming SACK advances the Cumulative TSN Ack Point, and the data sender is not in Fast Recovery. Only when these three conditions are met can the cwnd be increased; otherwise, the cwnd MUST not be increased." The current QUIC recovery text tries to clarify what fully utilized means, particularly in the presence of pacing.
I'm quite familiar with RFC 4960, and it does not have anything that is particularly helpful here. NAME I didn't see an fq-quantum in the code, but I could also be missing something.
Thanks for this -- I think it's the right direction. Some comments to try and herd this text some more.

Discussion:
7.9. <del> A sender becomes idle if it ceases to send data and has no bytes in flight.  A sender's congestion window MUST NOT increase while it is idle. </del> <ins> A congestion window that is under-utilized SHOULD NOT be increased in either slow start or congestion avoidance.  This can happen due to insufficient application data or flow control credit. </ins> <del> When sending data after becoming idle, a sender MUST reset its congestion window to the initial congestion window (see Section 4.1 of RFC5681), unless it paces the sending of packets.  A sender MAY retain its congestion window if it paces the sending of any packets in excess of the initial congestion window. A sender MAY implement alternate mechanisms to update its congestion window after idle periods, such as those proposed for TCP in RFC7661. 7.10. The congestion window should not be increased in slow start or congestion avoidance when it is not fully utilized.  The congestion window could be under-utilized due to insufficient application data or flow control credit. </del> <ins> A sender MAY use the pipeACK method described in section 4.3 of RFC7661 to determine if the congestion window is sufficiently utilized. </ins> A sender that paces packets (see pacing) might delay sending packets and not fully utilize the congestion window due to this delay.  A sender should not consider itself application limited if it would have fully utilized the congestion window without pacing delay. <ins> Bursting more than an intial window's worth of data into the network might cause short-term congestion and losses.  Implemementations SHOULD either use pacing or reduce their congestion window to limit such bursts. A sender MAY implement alternate mechanisms to update its congestion window after periods of under-utilization, such as those proposed for TCP in RFC7661. </ins> 8. 8.1.