Change:
reordered ack-eliciting packet.  This extension modifies this behavior. <del> If the endpoint has not yet received an ACK_FREQUENCY frame, or if the most recent frame received from the peer has an "Ignore Order" value of "false" (0x00), the endpoint MUST immediately acknowledge any subsequent packets that are received out of order. If the most recent ACK_FREQUENCY frame received from the peer has an "Ignore Order" value of "true" (0x01), the endpoint does not make this exception.  That is, the endpoint MUST NOT send an immediate </del> <ins> If an endpoint has not yet received an ACK_FREQUENCY frame, the endpoint immediately acknowledges any subsequent packets that are received out of order, as specified in Section 13.2 of QUIC- TRANSPORT.  An endpoint, that receives an ACK_FREQUENCY frame with a Reordering Threshold value other than 0x00, MUST immediately send an ACK frame when the packet number of largest unacknowledged packet since the last detected reordering event exceeds the Reordering Threshold. If the most recent ACK_FREQUENCY frame received from the peer has a "Reordering Threshold" value of 0x00, the endpoint does not make this exception.  That is, the endpoint MUST NOT send an immediate </ins> acknowledgement in response to packets received out of order, and instead continues to use the peer's "Ack-Eliciting Threshold" and "max_ack_delay" thresholds for sending acknowledgements.

Discussion:
I don't have a strong opinion on a varint vs 7 bits. I suspect 7 bits is more than enough, but I could go either way.
A requirement to send one ACK for at least every hundred data packets (or so) seems to have little impact on the processing cost nor capacity of the flow. I'd be concerned if any receiver did not do this. So, I still 7b think this actually is reasonable size.
I'm not so sure. MsQuic tunes this up to pretty large values for extremely high throughput, low loss scenarios. I'd rather not restrict things unnecessarily, unless you have a clear reason why we must not use a value so high?
On varint vs. 7-bit, it may be worth using a varint just for consistency and maybe to maintain the ability to completely ignore reordering by just setting a very high number.
There's still more work to do on this section, but in the interest of making the change, we're going to merge this and fix it later.
Per , the recommended value of packet reordering threshold is 3. That means that with vanilla QUIC v1, the sender would notice a single packet loss without any ack delay being introduced by the receiver. That's because when the receiver notices a gap, it sends an ACK immediately, then after receiving two more packets, it sends another ACK immediately. With the ack-frequency extension, this behavior changes. When the receiver notices a gap, one ACK is sent immediately. But after that, the next ACK will not be sent until the expiry of one of the thresholds being declared by the ACK_FREQUENCY frame. To give an example, let's say that Ack-Eliciting Threshold is set to 1/8 of CWND, and Request Max Ack Delay is set to 1/8 of RTT. Then, when there is one packet loss (or one range of packets being lost), the sender cannot notice that packet loss based on the packet reordering threshold until 9/8 RTT after the packet(s) was being sent. Is this going to be an issue for QUIC performance? I ask this because, compared to TCP, many have highlighted the improved efficiency of recovery in QUIC as one of the benefits.
I recall discussing something of sort in and , but I'm not sure if we considered these side effects enough.

IIRC, we abandoned the discussion to replace Ignore Order bit with Reordering Threshold, partly because we could not find a design that is easy to implement. But now, I think I might have a solution. We could: Replace the Ignore Order field with a Reordering Threshold field. State that receiver SHOULD ack every 2 packets that it receives, whenever the number of packets contained in the highest ACK range is no greater than that threshold. A sender that uses a threshold of 3 can advertise Reordering Threshold of 3 for example, to expect an immediate ACK to be sent when the receiver receives 3 packets after a gap. Or, a sender can advertise 5 instead, so that the loss would be noticed at an early moment even when there is an ACK loss. WDYT?
This is exactly what I was trying to mitigate with , but I think you explained the issue better than I did. With the current algorithm, QUIC would end up falling back to the time threshold loss detection, opposed to packet threshold, so it wouldn't be that bad, but there could be a delay. I'd like to agree on the properties we want. I think we want to ensure either time threshold or packet threshold are able to be triggered as quickly as possible. Given that, I think we need to send one (at least relatively, see last paragraph) immediate ACK upon observing out of order packets. That ensures that if only one packet arrives after the missing packet, it is detected in a timely manner by time threshold. Second, I think we want to avoid sending any other ACKs except the one which will trigger packet threshold. So the slightly complex bit is getting the algorithm for the second part correct. If you agree with the above goals, I think we need something a bit more complex than what you proposed. ie: What if the reordering threshold is 5 and you receive LostPN + 1, then LostPN+3, so there are already multiple ranges between the packet that could be declared lost and the reordering threshold. I'll try to describe and algorithm for that second ACK: Track the largestacknowledgedsent in an ACK When there are missing packets in the range , send an immediate ACK.

Additionally, I think we should point out that small scale reordering(ie: 1 packet or twiddles) are the most common, so it makes sense to process an entire batch of available packets before determining whether to send an immediate acknowledgement.
I see, sorry for revisiting the issue. I hope we'd resolve the issue correctly this time. Makes sense. The properties that I had in my mind were slightly different from what you have: send ACKs no more than vanilla QUIC v1 send ACKs to make sure that losses are detected as early as vanilla QUIC v1 provide opportunity to deal with ACK-losses, so that even when there are ACK losses, losses can be detected as early as vanilla QUIC v1 The reason I would not mind sending two or three ACKs to indicate one gap is because packet losses are rare. Let’s say that average packet loss ratio is 1%. Then, assuming Reordering Threshold of 3, the debate is between if the number of ACKs being sent in response to gaps would be ~1% of the packets that the sender sends, or ~2% of the packets that the sender sends (in case of the algorithm described in URL). I must admit that the number of ACKs used for indicating one gap increases relative to the Reordering Threshold, but I’m not sure if we want to use values above 3. All that said, I would be fine with the properties you describe, as I think the required algorithm is going to be simple enough, and because ACK loss is not a big problem assuming that packet losses are rare in general. Please Ignore my comments below this line, I see the algorithm being corrected in the comment above :+1: Though I think there's a tiny bug in the algorithm being proposed: I think this should be something like: When the new largest acknowledged is : , send an immediate ACK. Otherwise, if an ACK is sent due to other reasons immediately after a gap, then that gap would not be notified at the earliest moment. One example is PN=255 being lost, then PN=256 received and ACK being sent due to Ack-Eliciting Threshold. When the receiver receives PN=257 and PN=258, then it should send an immediate ACK, assuming Reordering Threshold of 3.
Thanks, I just updated my suggested algorithm, and I think it now matches yours, but is stated differently?

We use adaptive reordering, so it's not uncommon for our threshold to be 10+, which is 5 extra ACKs. Also, the every 2 option still requires the time threshold detection if you get one too few packet, whereas you could send an ACK at exactly the right threshold and the peer would immediately declare a loss. Not a big deal, but it's slightly suboptimal.
+1 to everything NAME says right above.
As I said on the mike during the WG meeting, there are two ways of distinguishing expected renumbering from packet losses. One is to check sequence numbers, and assume packet loss if >N packets received after hole. Another is a to check a reordering delay, if >T passes and the hole is still there. What I found in practice is that reordering delay thresholds work better than reordering number thresholds. Would be nice if the PR allowed for specifying a reordering delay, independent of the already specified ACK delay.
So let me understand the goal: you want to force an acknowledgment if a gap persists. A gap appears because a packet with a larger packet number is received. Then: Time: At that time, start a timer for some time less than the maximum acknowledgment delay and send an acknowledgment if the packet is still missing when the timer pops. (I'm not sure how to do this without having a timer running for every missing packet.) Number: This only requires that N packets after the gap arrive. I think that I understand Ian's algorithm for this.
In practice, you check the time of arrival of the following packets, and if the gap-delay is exceeded you fire the ACK.
NAME I think the point here is that the receiver has to remember when each packet was received, in addition to the packet numbers. That means that QUIC stacks have to change to the design of their ack queue, which makes this extension something much harder to implement than what it is now. At the moment, this extennion does not require changes to existing data structure. Separately, I am not sure how valuable it is to specify a signalling mechanism for time-based gap detection. ACKFREQUENCY frame already has Request Max Ack Delay field for time-based signalling. Senders can set this value to a fraction of RTT (e.g., 9/8 RTT) to induce ACK earlier than when the senders' time-based loss detection should be kicked.

Such approach does have the downside of Inducing ACKs regardless of having gaps. But IMO, that is not a big issue, because: Most congestion controllers require ACKs to be sent within a fraction of RTT. Reducing the number of ACKs from one per two ack-eliciting packets to one per fraction of RTT is important (because the reduction of ACKs by their numbers is huge), but going below that is not important. Considering these aspects, I am personally happy with only_ having the packet-number based gap reporting mechanism as proposed by Ian in .
NAME I suppose that there is some equivalence between "wait for N packets" and "wait for T milliseconds". If I had to not use a time delay, I would set N = CWIN*T/RTT. Or vice versa. After all, the scope of the reordering is either a fraction of the CWIN or a fraction of the RTT, depending how you look at it. I get the feedback that reasoning on packet numbers is easier for several implementations, so OK. Let's just use packet numbers.
I've looked at the pull request and I think that the technical mechanism is sound. However, it took me a long time to get there using this discussion and the proposed text. The explanation needs a lot of work. I'll break this into several comments. The definition NAME uses is correct, but it is very much non-obvious from the explanation. More text is needed in the pull request in order to explain why this is the case. The goal here is to ensure that a new acknowledgment is sent that includes a (new) largest acknowledged that is more than the reordering threshold past a gap. An acknowledgment like that will cause the packets in that gap to be declared lost. That is why you look for a gap before (note: [^1]). However, you don't want to send an immediate ACK if the other side already has actionable information. If the gap is before , there is no point in sending an ACK because the sender will have already declared those packets lost. All of that needs to be clearly articulated. I would also call attention to a critical difference between this and the ack-eliciting threshold. The ack-eliciting threshold counts unacknowledged packets, without paying any attention to their packet number. If there is a massive gap, that won't care.

The reordering threshold does care about packet numbers. That leads to the point that the two values are not related in any way. Either one can be larger with good reason, depending on how the sender is operating. Now we get to the harder bits of feedback... This change highlights how there are two piece of logic in play: The logic that determines when to send an immediate ACK in the presence of apparent reordering. In , this occurs when any gap appears. That equates to a recommended reordering threshold here of 1 (I think). The logic that determines when a packet is lost when processing ACK frames. In kPacketThreshold = 3. This document sets both to the same value and provides no way to separate the two. I think that's OK, but it is worth noting. As a matter of principle it seems tolerable for the former to be smaller than the latter, as we did in the original RFCs, but it is worth aligning the two on the basis that getting out of sync can lead to immediate acknowledgments that don't directly help with loss recovery. Again, I think that it is worth discussing this change, so that it is clear that the change is deliberate and to help readers understand what is going on. I can't see any functional difference between a reordering threshold of 0 and 1. Is that right? Could we benefit from changing the encoding to eliminate this redundancy? (I don't think we need to here - neither value is nonsensical - but I wanted to double-check.) Finally, I think that some examples would do a lot to improve comprehension. [^1]: Please don't call this , it's confusing: the logic here is going to decide whether this becomes the largest acknowledged packet; until then, it is the largest received packet (number).
NAME I think your point about the two different values should be added. And the logic in PR also needs to be clarified. But maybe we can just merge PR for now and then start another PR?
I think some wording clarifications would improve the text.