Change:
Adds a SHOULD NOT and clarifies that persistent congestion is across PN spaces. I was thinking that this is how it would be implemented, but I suspect this'll end up as a design change.
So a slight rewording might help, but you would have to disqualify losses associated with packets sent prior to getting an RTT estimate, as noted. But then RTT estimates are poor to start with (the first packet to establish a NAT binding can add noticeable delays, for instance).
Can we just forbid persistent congestion detection for the Initial and the Handshake PN space? That would be easier to reason about than the RTT estimate you had a while ago, when you sent the first packet.
NAME I'm sot sure if that would be sufficient as a fix, because packets belonging to ApplicationData PN space are also sent before an RTT sample is obtained (e.g., 0-RTT packets, 0.5-RTT data).
Right, this is what is about. Maybe we should say that only packets sent after handshake completion should be considered for persistent congestion detection?
NAME I agree that using handshake completion (or confirmation) would be better than status quo, though it is still questionable if that's good enough. IIUC, there are two issues: there is no continuity between initial RTT and real RTT the quality of RTT estimate is bad during the early stages of a connection Using handshake completion (or confirmation) as the signal concentrates on fixing the former, but does not address the second concern. If we are to make a design change, I think it would be beneficial to spend some time looking into addressing both of the issues. PS. Note that PTO taking RTTVAR into account does not work as a counterargument here, because we are using RTTVAR in an exceptional way (i.e. to estimate PTO of the past).
Yeah, this is tricky. Can we move the discussion to the issue please?
Based on the discussion we've had, don't we need a more comprehensive set of changes that deal with persistent congestion, sending probe packets, and calculating RTT?
NAME I think that depends on the following two aspects: First, the accuracy we want to have in the definition of persistent congestion (see URL). If we think of persistent congestion as a ballpark figure that resembles a blackhole period of somewhere around 3 PTO, during which an endpoint would have sent at least 2 or 3 packets, then I think the change proposed in this PR is sufficient. Second, if your concern is about endpoints not arming PTO for ApplicationData and that becoming an edge of persistent congestion, then I'd assume that the wort case that we might want to discuss is when all the following conditions are met: the TLS handshake transcript send by the server does not fit in 3 datagrams (therefore the server would have RTT estimate when sending the first 0.5 RTT data) client spends long time in verifying the certificate chain both the 1-RTT packets that carrried 0.5 RTT data and the first 1-RTT data (i.e. the one containing HANDSHAKE_DONE) gets lost Under such scenario, persistent congestion would be declared. But then, the condition matches the definition of the previous paragraph. Therefore, I think we can call this a non-issue, if we are fine with what's written in the previous paragraph.
I think we might need to do something to avoid or at least caution against terrible RTT samples, but that's being discussed in
This is separable from , and I wouldn't mix the two. I agree with NAME that the design issue here is about whether 3 x PTO is about time, about packets, or both. That needs consensus. This PR is good if consensus is on the issue is that PTO is (mostly) about time. We should discuss that on .
NAME : That's probably a good idea.
I added a few lines of pseudocode and a comment. Hopefully that's helpful.
FWIW this is what mvfst (and perhaps others?) has been doing since persistent congestion was implemented, since at the time it seemed silly to calculate the period using the initial RTT. However, at the time we still had the handshake timeout and so persistent congestion did not even apply to initial and handshake packets (see ). More recently our deployment has relied on PTOs for the handshake and thus this check, and there are no obvious problems with it on the Internet. I am in favor of this change even if it is perhaps not exhaustive in its coverage of unlikely pathological cases.
My reaction is the same as NAME I had to think a few times over to make sure that this was adequate, and I believe it is. It covers an egregious case explicitly, which is important.
I'm merging this so Jana can pull any conflicts into
A sender sends a tail of 10 packets, packet number 1 to 10. It now receives an acknowledgement for packets 5 and 10, which establishes the loss of all other packets. It's not clear how to perform the check for persistent congestion now. Several options come to mind: check if the period between 6 and 9 is longer than 3PTO check if either the period between packet 1 and 4 or the period between packet 6 and 9 is longer than 3PTO The ACK doesn't establish loss of all tail packets, so there's no persistent congestion. In an offline discussion, both NAME and NAME expressed preferences for different options here. I'm not sure what the right thing to do is here, but at least the draft should be clearer on what exactly establishes persistent congestion and what doesn't.
Consider the case where the sender receives an acknowledgement for packets from 5 to 10, and the period between 1 and 4 was longer than 3*PTO. IIUC, this is clearly a persistent congestion. The original example is strictly worse than that, and I think it would make sense to declare persistent congestion in the case too. In other words, I think that 3 is incorrect. That said, I do not have a strong opinion regarding if we should specifically recommend either of 1 or 2, considering that the logic (or the time threshold) being used to detect persistent congestion is merely a recommendation.
FWIW, we had some discussion regarding the design rationale behind persistent congestion in , and it seemed to me that it was mostly about the amount of time the sender cannot make any progress.
seems also related
I think we should keep it simple, which IMO is do nothing. Or if we want to add a recommendation, we can say that you start with the largest lost packet and try to detect Persistent congestion until either you have found 1 occurrence or reached the end of list.
This is fixed in , and is an editorial fix.
Consider the following case: endpoint sends Initial after 1 second, endpoint sends Initial (PTO) after 1ms, the endpoint receives an ACK for Initial (PTO) At this point, the endpoint would declare persistent congestion because the duration of congestion (1 seconds) is greater than the threshold (i.e. where smoothedrtt is 1ms, rttvar is 1ms, and maxackdelay is zero). The problem is that determining persistent congestion based on SRTT is not a good idea until the RTT estimate becomes stable. I think the easiest fix (which is beneficial at this late stage) would be to disable persistent congestion until the handshake concludes. [[Spawned off from as a dedicated issue]]
My interpretation here is that the single packet loss does not result in persistent congestion as it spans precisely one packet. That changes of course if there are two packets sent on PTO and the first of those is lost as the losses span multiple packets over a longer period.
NAME is right. The example should be: endpoint sends Initial (PN 1) after 1 second, endpoint sends two Initials on a PTO (PN 2 and PN 3) after 1ms, the endpoint receives an ACK for PN 3 Now we have a loss spanning over 1s, which is >> PTO.
I don't think persistent congestion can be enabled until there is an RTT sample, because smoothed_rtt and rttvar don't have measured values. I wrote an editorial PR to that effect, so assuming that's sufficient, this is an editorial issue.
NAME Thank you for the PR. As I've stated in the PR review comment, per my understanding, the problem is about applying the first RTT sample to the probes sent before obtaining that RTT sample. Because the difference between the first RTT sample and initial RTT could be huge, using the first RTT sample to estimate how many probes have been sent based on the initial RTT does not work well.
I agree that the PR doesn't solve this problem. Conservatively, you could avoid using losses of any packets that were sent before an RTT sample was received as legitimate for persistent congestion, and this might not be particularly hard to implement. You need one variable that is set on the first RTT measurement (largest PN sent for each space), and ensure that all PNs considered when detecting persistent congestion are greater than that number (per PN space). Not doing persistent congestion before the handshake would be quite an aggressive departure from what our intent is.
I agree with Jana that disabling persistent congestion before the handshake would be quite aggressive and quite different from TCP, so I'd prefer to avoid that. I was thinking persistent congestion was a timer that would be started when a packet is sent, but it's a time interval, so I agree the PR is not sufficient as is. I'll update it along the lines of Jana's suggestion, which is what I intended.
So, I understand the disconnect now after discussing with NAME This issue is not just about before an initial RTT sample, it's about the case where the network RTT were to similarly reduce later in the connection. In such a case, when loss also happens, the sender could retroactively declare persistent congestion based on the new RTT estimate, while the previous RTT estimate might have caused fewer than 3 PTOs to fire. Specifically, the sender might have sent fewer than 3 packets as PTO responses into the network before marking the network as having persistent congestion. This seems like a corner case to me that may not be necessary to fix (beyond what is being fixed in ). But I want to hear others' opinions.
NAME Thank you for summarizing the issue. Yes, my point was that the current definition of persistent congestion is inaccurate if it based on the rationale that the endpoint would have had occasions to send 3 probe packets, because the endpoint might have had a larger RTT estimate before receiving the ACK that triggers the persistent congestion check. That said, in the current computation, we do have the guarantee that the endpoint would have sent at least 2 probe packets, because PTO never decreases by more than 25% when obtaining just one RTT sample. If we think that is enough, I'm fine with going with what we have now in .
Thanks for reading the updated PR. Assuming we go with something along the lines of , I think this ends up as a design change, unless others think this is just a clarification of existing text?
I think that this issue is a design issue and also that the current state of is a design change.
This is a design issue. Based on what NAME notes in URL, the design issue here is about whether 3 x PTO is about time, about packets, or both. That needs consensus. As it stands, if it's mostly about time, then fixes this issue. I am happy with that fix.
My understanding was the point of changing from PTOs to persistent congestion was to make it about time. If we think the change in RTT caused by the ACK which triggers the persistent congestion check is an issue, we should ignore that sample when doing the determination. But I'm not convinced the extra complexity is worth it.
NAME NAME Thank you for sharing the rationale behind the design. I am convinced that the fix in is fine.
is now merged, but I realized that it has a slight problem. The text and the pseudo-code don't agree. Since this has gone through consensus, I'm continuing this discussion on .
The wording around persistent congestion sounds across pnspaces: But DetectLostPackets function is per pnspace and persistent congestion checking is invoked from there. So if DetectLostPackets find all packets of one pnspace within a persistent congestion window are lost but packets from other pnspace within this time window are not lost, is this persistent congestion?
I think it should only be for ApplicationData, and say that we expect the handshake timeout to be smaller than the persistent congestion threshold, so it does not need to apply to Initial or Handshake spaces. Prior to the persistent congestion concept, RTOs(and then consecutive PTOs) were used as a signal of persistent congestion, and the RTO and PTO alarm are only active when the crypto handshake alarm is not, so I think saying it's ApplicationData only is keeping with the original design.
Oh i see your point. Basically crypto timer would have been fired earlier than persistent congestion. So this is for AppData only and will apply to AppData only anyway, then I shouldn't have to worry about packets from more than on pn_space are in the persistent congestion window.
It took me quite a while to confirm that "is first rtt sample" is good. It's slightly more expansive than what I had implemented (which records This has the effect of including packets sent prior to getting that sample. But as this is only packets sent within a single RTT, it's guaranteed to be less than a PTO.

Discussion:
The RECOMMENDED value for kPersistentCongestionThreshold is 3, which is approximately equivalent to two TLPs before an RTO in TCP. <ins> The persistent congestion period SHOULD NOT start until there is at least one RTT sample.  Prior to an RTT sample, the duration cannot be correctly calculated.  Waiting for one RTT sample also avoids spuriously declaring persistent congestion when the initial RTT is larger than the actual RTT. </ins> This duration is computed as follows: For example, assume: