Original text:
other packet number spaces with in-flight data, coalescing packets if possible.  When the PTO timer expires, and there is new or previously sent unacknowledged data, it MUST be sent.

Reviewer comments:
Indicates that peers can skip a packet number when sending a single PTO packet to elicit a faster ACK.
This is a clever idea, but I don't think that we want to write it down in the spec. It's allowed by the spec, so people can surely do it, but we are talking about immediate ACK signaling in other ways too, and I don't think we want to encourage this behavior.
NAME Yeah, it's incredible how much money they will make by skipping packet numbers. If I had just known about this before...
I'm going to run an experiment comparing 1 PTO packet, 2 PTO packets, and 1 with skipping a packet number and report back the results. If the results are good enough with 1PTO + gap, we can consider whether this is an optimization worth mentioning.
Can also test this both sides at the same time? I'm still concerned this might run away in ACK floods.
Google and Chrome have been running experiments with the new IETF QUIC PTO algorithms post-handshake for a few months with Google QUIC, and it appears the first PTO is too conservative relative to Google QUIC. On the positive side, the rate of spurious TLPs/PTOs is MUCH lower, so that's WAI. This slows down detection of tail losses and causes a regression in 95%(and higher) latency metrics. It's also visible as an increase in rebuffers for YouTube. The changes aren't enormous, but they're large enough I can't launch this as is. I believe it's possible to make the first PTO slightly earlier, and stay aligned with the principles of TCP TLP and RTO. Specifically, my suggestion is to set first PTO expiry from the left edge(), not the right edge, but there's an additional safety measure of waiting 1.5 * SRTT since the last ACK-eliciting packet was sent(1.5 is from TLP). It'll be a few more weeks before we have data from Chrome Stable, but I believe this may be enough to remove the regression. Chromium code . One other note is we found that only sending one PTO packet seems sufficient as long as a packet number is skipped, so the delayed ack timer doesn't come into play. (Later added in PR ) NAME or others may also have production metrics relevant to this issue.
is entitled "IP Router Alert Considerations and Usage". Do you refer to (5.1) in ? That reads to me like a left edge condition, based on "timer is not running". Whereas : I can see one 1.5sRTT factor in , but that is inflated quite a bit, so I'm not sure exactly what you think motivates this 1.5 value. I do see the reason for your caution here. You want to probe based on the first unacknowledged packet, but leave a buffer relative to the last unacknowledged packet (assuming all elicit acknowledgment, of course). This is because all outstanding packets will all cause an acknowledgment if they arrive. So this seems sensible, if a little more aggressive than we have been. I don't know what the usual relationship between RTT and RTTvar is - I'm guessing that like everything it varies greatly. I'm inferring that the buffer you want is as much based on allowing for variance as much as anything else. The PTO is chosen to correspond roughly to 4-sigma, which is 99.994% and probably too conservative. That conservatism is probably what dominates your numbers. That also suggests that in the cases that drive your numbers, 4rttvar is greater than 0.5rtt. I'm guess that it is a LOT greater. Possible alternatives would be to cap rttvar to a proportion of the RTT estimate, maybe just for th purposes of calculating the PTO. 12.5% would get you almost exactly what you describe without changing the PTO calculation. It seems to me that PTO on 2-sigma and capping rttvar to RTT/4 is probably better. That's 2-sigma/95% confidence without the cap, and exactly your numbers if the cap has to be used. Either way, that recognizes that one probe timeout is primarily dependent on the estimated RTT and is less susceptible to variation in that estimate. I would like to request that you open a separate issue for the packet number skipping thing. That seems separable.
Yes, I meant RFC 6298, and I just updated the original post to reflect that. In terms of rttvar, the production data I'm seeing indicates rttvar is fairly close to 1/2 SRTT on average, which isn't so surprising given SRTT is an EWMA and for it to be on the order of SRTT, it would require estimates fairly close to 0 to balance out the higher estimates. Unfortunately, rttvar is RTT variation, not RTT variance(or RTT stdev). As such, I don't know if we can reason about what fraction of spurious PTOs would be within SRTT + 4RTTVar, even if we could assume a normal distribution(which is also likely wrong, but I think the least problematic assumption at the moment). I'd really rather not cap rttvar in any way, since I can't come up with any intellectual justification for that. The 1.5 constant from TLP does seem a bit arbitrary, but it is larger than SRTT. We added code in Chrome to try RTTstdev(EWMA'd), but that's even more experimental than the suggestion in my first post on this issue. I think there's a good chance 2RTTStdev is the best option(vs 4*RTTVar), but I don't have data to support that yet. I reopened PR for packet number skipping. I can open an issue if needed, but it's a simple non-normative PR.
Ah, I slipped into the same trap as usual. Mean absolute difference from the mean is the EWMA approximation we are using. Mean absolute difference is always less than standard deviation, with the factor being ~0.8 for a normal distribution. Do you have information on the distribution of RTT values that might help determine whether 2 is the right multiplier? If you have a normal distribution or something approximating that, then 3 would be closer (3.2 and change even). I can justify capping PTO, but it requires that you be comfortable with being more aggressive. But, at least to me, it seems like this is all about more aggressive sending. Which naturally improves performance most of the time. The network is likely to be able to absorb a minutely higher rate of probe packets as a result. Based on recent history of the RTT estimate, the peer can produce an acknowledgment within sRTT most of the time. Anything more than value is included only to allow more time for the peer to respond. However, if this is the trailing edge of a flight of packets, then any packet in that flight can be acknowledged. Allowing for some amount of variation in delays relative to the leading edge seems sensible, but delays on the trailing edge do not need any such justification. The only reason not to use the leading edge alone is that you don't want to send probes when you are sending productive data within the congestion window. On that basis, probing on the RTT estimate is fine. That means dead air of one RTT, and maybe some wasted probes because of jitter in the network that extends the effective round trip time. But you don't need to avoid unnecessary probing with very high confidence, you just need to avoid it most of the time. The risk with probing sooner is that RTT might genuinely increase, so that probes - which might exceed the congestion window - don't help at all. That should be rare; we have to treat past performance as a predictor for future gains, that's the basis of all of these systems. As long as there is an exponential backoff on PTO, we're unlikely to cause significant congestion as you only ever send a handful of probes before flipping to persistent congestion. A shorter and more aggressive timer here also results in hitting persistent congestion earlier. So if there really is a routing flap that increases RTT significantly - a case where you really want to reset the congestion window anyway - then you will just end up in hitting that state sooner. Another justification for a shorter PTO is that your estimation disadvantages shorter sequences of packets. A single packet flight doesn't get probed any sooner under your algorithm, but that is more likely to see loss of the entire flight than a longer sequence of packets.
Out of curiosity, is there any ongoing work using perceptrons or similar for predicting retransmission behaviour like it is being used in CPU branch prediction? URL URL
NAME Everything you said makes sense to me. I'm hoping to come up with a proposal that does 2 things: 1) Is not a regression for our current gQUIC implementation (or TCP best practices) 2) Build on well publicized and accepted concepts from TCP(ie: TLP and RTO) 3) Reduce the rate of spurious retransmits vs gQUIC. In terms of RTTVar vs an EWMA'd Stdev, I suspect the end result may be something like: SRTT+4RTTVar OR SRTT + 2(or 3)EWMA_Stdev, but I don't have enough data to know for sure yet. The experiments have been started, so hopefully I should know more in 2 weeks.
Instead of 1.5 SRTT we could consider SRTT + RTTVar. I think that's slightly more principled, but should be similar to 1.5 SRTT for typical connections.
I have experiments ongoing, but they're waiting on a Chrome rollout, so I'm happy with this being parked or otherwise not delaying WGLC.
I'll park this for now. Please propose to unpark once you have data.
I think this is worth exploring. We can also run an experiment with a subset of our traffic, so that we can be sure this is at least neutral for our current deployment. I don't think we should ignore a major deployment's signal, as this will inevitably lead to people shipping this (or some other mitigation) eventually as the de facto spec, similar to various TCPisms that are de facto standards.
If we want to have this impact QUICv1, we need data and a proposal in a relatively short timeframe. We're very close to being able to last-call the specs, and I don't want to delay that. So interested parties should prioritize this.
The most recent set of experiments indicate that my above suggestion of arming for max(leftedge + SRTT+4RTTVar, rightedge + 1.5SRTT) is a very slight improvement for some metrics, but it looks to be in the noise for most metrics. We did find SRTT+2RTTVar is as good or better than SRTT+4RTTVar for all application metrics, with a very small difference in retransmit rates(probably because PTO can send new data). I'd be very curious to see what NAME finds in his testing, but I'm increasingly inclined to close this with no action. Update: The above being said, we're going to default enable the version that arms from the left and right edges for the first PTO.
I will have data on this in a few days.
The editors discussed this and concluded that it would be best to mark this QUICv2.
Following up, I ran a test for 4 days on a major country. The results are mostly in line with what NAME saw. Tail response latency (p95) saw a measurable but small decrease of about 1 percent. Various application-level outlier metrics (e.g. time to load images, time to load a feed, etc.) showed slight improvements that also correspond with the change. There were no regressions in any other metrics. Based on this I think the strategy of arming the first PTO based on max(leftedge + SRTT+4RTTVar, rightedge + 1.5SRTT) is indeed an improvement, but it is not huge. I would agree with punting work on this to v2, since we plausibly will have time to come up with different strategies here that may be a larger improvement.
Thanks for the data NAME !
This probably shouldn't make me nervous, but it still does. But I still greatly appreciate the data. For my own peace of mind (though you will never alleviate the nervousness), can you confirm that you track both observed loss rates AND the rate of spurious retransmission? It strikes me that there would be high correlation between improvements in latency and real loss, so the loss rates probably don't change except where the faster transmission coincides with true congestion (which I would expect to be undetectable), and the same is likely also true for spurious retransmission, but I just wanted to confirm.
NAME For what it's worth, I did extensive testing of this change on smaller scale first, e.g. A/B tests on groups of individual hosts in different locales to ensure that nothing was fundamentally problematic. But indeed, there is good reason to be nervous when experimenting with recovery on the Internet One thing I'll note here is that the vast majority of the newly-fired PTOs in this test (i.e. PTOs that previously would have fired later) were not retransmissions at all but rather PTOs using fresh application data. As such the retransmission rate, spurious or otherwise, was pretty much constant when compared to the control. I think your intuition about the loss rate is totally warranted; in this test though there was not a statistically significant change in the observed loss rate, either the average or in the p95+ or p99 tail.
Marking as proposed and closing.

Revised text:
other packet number spaces with in-flight data, coalescing packets if possible. <ins> If the sender wants to elicit a faster acknowledgement on PTO, it can skip a packet number to eliminate the ack delay. </ins> When the PTO timer expires, and there is new or previously sent unacknowledged data, it MUST be sent.