Change:
I started with trying to do small fixes, but quickly found that restructuring a bit and clarifying the details of what the sender was precisely supposed to do was the better than trying to be terse. This adds a couple of normatives, which may be unnecessary, and I'm happy to take suggestions on those.
It would be necessary to use much resource to implement the detection of persistent congestion described here because of "across all packet spaces". So, let me ask a basic question. What is the motivation to define persistent congestion across all packet spaces? Isn't it good enough to limit persistent congestion only to 1-RTT?
NAME I'm not sure if the amount of state an endpoint needs to retain changes depending on if persistent congestion is a connection-wide event. FWIW, we had some discussion regarding the complexity in , and my take is URL
I have implemented this algorithm based on the pseudocode in . As I said in URL, it was not difficult for Haskell quic to implement this since the data structure was able to be reused. However, I'm not sure if it is easy to implement this algorithm for all implementations and benefits outweighs its complexity.
NAME let's merge this one. Then we can talk about the pseudocode in .
This resolves a design issue, so flagging it so that it doesn't get merged accidentally.
NAME : This is part of the consensus call, so we need to wait for that to complete.
We resolved with , but I realized that it has a problem. The text and the pseudo-code don't agree. The text says: Since the starts with a packet that was sent in the past, this statement means that persistent congestion only includes packets that were sent after an RTT sample was taken. The pseudo code says: This says that the persistent congestion can only be established on the second (or later) ack that yields an RTT sample. This allows for packets that were sent before the first RTT sample was taken to be included in the persistent congestion period. I had convinced myself that the pseudo-code was adequate, but I missed the fact that the text was not aligned.
I see two options. Option 1: Implementing what the text says will require a new variable that records the next PN when the first RTT sample is taken. Option 2: Changing the text to match the pseudo-code will look something like this:
An example might help: 1: Pkt 10, 11, 12 sent 2: PTO, Pkt 13, 14 sent 3: PTO, Pkt 15, 16 sent 4: Ack 10 received. 5: Ack 16 received. At 5, the pseudo-code would declare persistent congestion if . The text however wouldn't, since there was no RTT sample during the persistent congestion period, t1 to t3.
Option 1 and 2 have a behavior difference with the following implication: With option 2, a sender might mark persistent congestion with fewer PTOs than if the sender had a more accurate RTT estimate. This is arguably not a significant concern, since: the sender will still have sent at least two packets in that duration and marked them lost, since you need two ack-eliciting packets to be lost to declare persistent congestion; and the number of packets sent is also sensitive to how the application sends, and a sender cannot rely on that anyway. Option 2 has the benefit that it is simpler to implement. On balance, I think Option 2 is better, though it does make this a design change.
Thanks for the example, I was unclear on the issue up until then, though I think a slightly stronger example might be: Ack 14 received, since that's only a single PTO. I think 1 aligns more with fully fixing the original issue, since it seems like your example is a case we agreed we would not want to declare as persistent congestion in . That being said, it makes the pseudocode more complex. Persistent congestion is a new concept, so I suspect implementations will vary anyway.
I agree that making the change to the pseudo-code seems to be the more correct solution. That being said, I'm not planning to implement persistent congestion detection during the early phases of the handshake (but only after confirmation of the handshake), since I don't see clear benefits of the added complexity of cross-packet-number-space persistent congestion detection.
NAME I personally think that's fine, though I'll note that TCP is less aggressive than that, which is why I think we should describe this as something that covers the handshake as well.
Given what I've heard, I think this is heading in an editorial direction(fix the pseudocode), though that needs a PR.
Yes, I'm not wedded to the design change, and am happy for this to be an editorial one. I'll write up a PR.
Should this become proposal-ready?
NAME : Since this is a pseudo-code issue, it should be an editorial change (the pseudo code is in the appendix). will fix this issue, but that is waiting on , which is included in the latest consensus call.
Persistent congestion has to start with an ack-eliciting packet. The section on persistent congestion isn't very clear about this point. This is because a non-ack-eliciting packet won't be followed up with any probes. It is therefore possible to send one packet and then not send anything else for an extended period. After this, the loss of just the next packet will cause persistent congestion. Note that persistent congestion can be caused by the loss of just two packets after multiple PTOs have occurred, but the common case is that you have at least three packet, or four if you send two packets on PTO.
Would it make sense to change the persistent congestion condition to require at least 3 packets to be lost, in addition to the time threshold that we already have?
FWIW we have had some discussion around this in , and it seemed to me that there were preference to not do that.
I implemented this as requiring ack-eliciting on the basis that once you are two PTOs out, it's not that bad that you hit persistent congestion, but I wouldn't object to that.
I can at least change this to ack-eliciting -- that's a simple editorial change. That was the intent anyway. If we want to change the way persistent congestion is established, that's a bigger issue, so let's separate that issue from this one.
My understanding is that it is not, but it's unclear.
It can't be, since the congestion window is a global property, and declaring persistent congestion results in a collapse of the congestion window.
It shouldn't be, because congestion is a network/connection-level property. That said, NAME and I were discussing this offline, and I'm now coming to terms with the fact that it's hard to do this globally across packet number spaces. Basically we have no current model to track packet transmissions across packet number spaces, and implementations do not typically do this either. I think this is hard to do across PN spaces... or at least, I've not been able to figure out a reliable way to do it. Based on what I've been able to come up with so far, I suspect we might want to disable persistent congestion until the handshake is confirmed. That is a departure from TCP, but I believe that is still ok. In practice, we have a congestion window reduction that occurs on losses in this period, and since the handshake will involve only a few packets in flight, losses should cause the cwnd to reduce to a small enough value. Thoughts?
NAME I think that it is actually easy; see URL
(NAME Could you move the discussion on that from the PR to this issue?)
NAME Sure. The following is what was written in URL One trivial way of implementing is as follows: Retain the entries in sentmap (the list of packets being sent) for at least 3 PTO. When receiving an acknowledgement, mark a hole in the sentmap for the packet being acked. When a loss is detected in a packet number space, check if that loss spans across more than 3 PTO, by traversing the sentmap. Then, if did, check the sentmap of other packet number spaces (by traversing through those sentmaps) to see if any ACK-eliciting packet sent in that period has been acked. This approach might sound too trivial, as it is O(N) where N is the number of packets being lost (or in case there are multiple PN spaces in action, the number of inflight packets in other PN spaces). But in practice, I think this approach is sufficient in terms of performance, because N would be like 3 for the current packet number space, and because we would not be sending that many packets on Initial and Handshake packet number spaces.
In , NAME states: It does not require additional state. In QUIC, there is an expectation that packet numbers would be used sequentially, starting from zero. Therefore, a "gap" can be detected when iterating through the sentmap and if the packet number increases (or decreases) by more than one. Note that endpoints are recommended to treat receipt of an acknowledgment for a packet it did not send as a connection error of type PROTOCOLVIOLATION, if it is able to detect the condition_ (). Therefore, when intentionally skipping a packet number, it is a good idea to have that recorded in the sentmap, and close the connection when the peer acks that skipped packet number. To paraphrase, it is possible determine if a packet has been acked, by looking for a gap between the packet numbers within the list of unacknowledged packets.
I don't think this is trivial, quite the opposite actually. A "hole" would be extra state (note that just the absence of a packet at a certain PN does not constitute a hole, since you could've skipped that PN to mitigate the optimistic ACK attack), and that additional state has to be cleaned up. In my implementation, when a packet is acknowledged, I know I can delete it and get rid of all state associated with it. Deviating from this, just to be able to do do persistent congestion detection across packet number spaces, seems like a lot of hassle to me. As NAME described, any packet loss during the handshake will half the congestion window, and with , it will even reduce the congestion window to half the bytes in flight (which will commonly be less than the cwnd during the handshake). Simple Reno-style congestion control will therefore lead to behavior that is very similar to actually doing persistent congestion detection across packet number spaces, without all the additional complexity.
There was a comment reorder. NAME I think your question has been answered in my previous comment. NAME Going back to the spec discussion, even if it is the case that some implementations might have difficulty in implementing persistent congestion detection covering multiple packet number spaces, that alone does not mean that our recommendation should deviate from what TCP does. I would very much prefer phrasing the discussion around if we should have a MAY allowing endpoints skip persistent congestion detection during the handshake.
There are different ways to implement packet number skipping for optimistic ACK defense, which don't involve creating fake packets in the sentmap. Those ways would make it impossible to use the "hole method" without creating additional state for already acknowledged packets. No matter what we do, our recommendation will deviate from what TCP does. Going back to the example you posted in URL, the acknowledgement of the Handshake packet would already trigger an RTO in TCP. In QUIC, we can't do that, because the packets live in different packet number spaces, and an ACK can only ever declare loss in its own packet number space. So the question cannot be if we want to deviate from what TCP does. We already do, and there's no way around it. Instead, we need to ask how much additional complexity we want to introduce to handle this corner case. I'd argue that the benefit is extremely limited, and we should therefore pick the simplest solution: only run persistent congestion detection for packets sent after handshake confirmation.
For some of the things, we have to deviate from TCP, because there is no way around it. But for this particular case, it is easy for some if not most of the QUIC stacks. Then, there's no ground to "recommend" something that deviates from TCP.
fixes this by clarifying that persistent congestion is across PN spaces. Since the resolution in introduces normative language, I would mark this as design.
A sender sends a tail of 10 packets, packet number 1 to 10. It now receives an acknowledgement for packets 5 and 10, which establishes the loss of all other packets. It's not clear how to perform the check for persistent congestion now. Several options come to mind: check if the period between 6 and 9 is longer than 3PTO check if either the period between packet 1 and 4 or the period between packet 6 and 9 is longer than 3PTO The ACK doesn't establish loss of all tail packets, so there's no persistent congestion. In an offline discussion, both NAME and NAME expressed preferences for different options here. I'm not sure what the right thing to do is here, but at least the draft should be clearer on what exactly establishes persistent congestion and what doesn't.
Consider the case where the sender receives an acknowledgement for packets from 5 to 10, and the period between 1 and 4 was longer than 3*PTO. IIUC, this is clearly a persistent congestion. The original example is strictly worse than that, and I think it would make sense to declare persistent congestion in the case too. In other words, I think that 3 is incorrect. That said, I do not have a strong opinion regarding if we should specifically recommend either of 1 or 2, considering that the logic (or the time threshold) being used to detect persistent congestion is merely a recommendation.
FWIW, we had some discussion regarding the design rationale behind persistent congestion in , and it seemed to me that it was mostly about the amount of time the sender cannot make any progress.
seems also related
I think we should keep it simple, which IMO is do nothing. Or if we want to add a recommendation, we can say that you start with the largest lost packet and try to detect Persistent congestion until either you have found 1 occurrence or reached the end of list.
This is fixed in , and is an editorial fix.
This is much clearer, thanks.One more suggestion

Discussion:
7.8. <del> When an ACK frame is received that establishes loss of all in-flight packets sent over a long enough period of time, the network is considered to be experiencing persistent congestion.  Commonly, this can be established by consecutive PTOs, but since the PTO timer is reset when a new ack-eliciting packet is sent, an explicit duration must be used to account for those cases where PTOs do not occur or are substantially delayed.  The rationale for this threshold is to enable a sender to use initial PTOs for aggressive probing, as TCP does with Tail Loss Probe (TLP; see RACK), before establishing persistent congestion, as TCP does with a Retransmission Timeout (RTO; see RFC5681).  The RECOMMENDED value for kPersistentCongestionThreshold is 3, which is approximately equivalent to two TLPs before an RTO in TCP. </del> <ins> When a sender establishes loss of all in-flight packets sent over a long enough duration, the network is considered to be experiencing persistent congestion. </ins> <del> The persistent congestion period SHOULD NOT start until there is at least one RTT sample.  Prior to an RTT sample, the duration cannot be correctly calculated.  Waiting for one RTT sample also avoids spuriously declaring persistent congestion when the initial RTT is larger than the actual RTT. </del> <ins> 7.8.1. </ins> <del> This duration is computed as follows: </del> <ins> The persistent congestion duration is computed as follows: </ins> <del> Unlike the PTO computation in pto, persistent congestion includes the </del> <ins> Unlike the PTO computation in pto, this duration includes the </ins> max_ack_delay irrespective of the packet number spaces in which losses are established. <del> The following example illustrates how persistent congestion can be established.  Assume: </del> <ins> This duration allows a sender to send as many packets before establishing persistent congestion, including some in response to PTO expiration, as TCP does with Tail Loss Probes (RACK) and a Retransmission Timeout (RFC5681). The RECOMMENDED value for kPersistentCongestionThreshold is 3, which is approximately equivalent to two TLPs before an RTO in TCP. This design does not use consecutive PTO events to establish persistent congestion, since application patterns impact PTO expirations.  For example, a sender that sends small amounts of data with silence periods between them restarts the PTO timer every time it sends, potentially preventing the PTO timer from expiring for a long period of time, even when no acknowledgments are being received. The use of a duration enables a sender to establish persistent congestion without depending on PTO expiration. 7.8.2. A sender establishes persistent congestion on receiving an acknowledgement if at least two ack-eliciting packets are declared lost, and: all packets, across all packet number spaces, sent between these two send times are declared lost; the duration between the send times of these two packets exceeds the persistent congestion duration (pc-duration); and a prior RTT sample existed when both packets were sent. The persistent congestion period SHOULD NOT start until there is at least one RTT sample.  Before the first RTT sample, a sender arms its PTO timer based on the initial RTT (pto-handshake), which could be substantially larger than the actual RTT.  Requiring a prior RTT sample prevents a sender from establishing persistent congestion with potentially too few probes. Since network congestion is not affected by packet number spaces, persistent congestion SHOULD consider packets sent across packet number spaces.  A sender that does not have state for all packet number spaces or an implementation that cannot compare send times across packet number spaces MAY use state for just the packet number space that was acknowledged. When persistent congestion is declared, the sender's congestion window MUST be reduced to the minimum congestion window (kMinimumWindow), similar to a TCP sender's response on an RTO (RFC5681). 7.8.3. The following example illustrates how a sender might establish persistent congestion.  Assume: </ins> Consider the following sequence of events:

packet 9 is received at t = 12.2. The congestion period is calculated as the time between the oldest <del> and newest lost packets: 8 - 1 = 7.  The duration for establishing persistent congestion is: 2 * 3 = 6.  Because the threshold was reached and because none of the packets between the oldest and the newest lost packets were acknowledged, the network is considered to have experienced persistent congestion. While this example shows the occurrence of PTOs, they are not required for persistent congestion to be established. When persistent congestion is established, the sender's congestion window MUST be reduced to the minimum congestion window (kMinimumWindow).  This response of collapsing the congestion window on persistent congestion is functionally similar to a sender's response on a Retransmission Timeout (RTO) in TCP (RFC5681) after Tail Loss Probes (TLP; see RACK). </del> <ins> and newest lost packets: 8 - 1 = 7.  The persistent congestion duration is: 2 * 3 = 6.  Because the threshold was reached and because none of the packets between the oldest and the newest lost packets were acknowledged, the network is considered to have experienced persistent congestion. While this example shows PTO expiration, they are not required for persistent congestion to be established. </ins> 7.9.