Change:
I don't have a strong opinion on a varint vs 7 bits. I suspect 7 bits is more than enough, but I could go either way.
A requirement to send one ACK for at least every hundred data packets (or so) seems to have little impact on the processing cost nor capacity of the flow. I'd be concerned if any receiver did not do this. So, I still 7b think this actually is reasonable size.
I'm not so sure. MsQuic tunes this up to pretty large values for extremely high throughput, low loss scenarios. I'd rather not restrict things unnecessarily, unless you have a clear reason why we must not use a value so high?
On varint vs. 7-bit, it may be worth using a varint just for consistency and maybe to maintain the ability to completely ignore reordering by just setting a very high number.
There's still more work to do on this section, but in the interest of making the change, we're going to merge this and fix it later.
Per , the recommended value of packet reordering threshold is 3. That means that with vanilla QUIC v1, the sender would notice a single packet loss without any ack delay being introduced by the receiver. That's because when the receiver notices a gap, it sends an ACK immediately, then after receiving two more packets, it sends another ACK immediately. With the ack-frequency extension, this behavior changes. When the receiver notices a gap, one ACK is sent immediately. But after that, the next ACK will not be sent until the expiry of one of the thresholds being declared by the ACK_FREQUENCY frame. To give an example, let's say that Ack-Eliciting Threshold is set to 1/8 of CWND, and Request Max Ack Delay is set to 1/8 of RTT. Then, when there is one packet loss (or one range of packets being lost), the sender cannot notice that packet loss based on the packet reordering threshold until 9/8 RTT after the packet(s) was being sent. Is this going to be an issue for QUIC performance? I ask this because, compared to TCP, many have highlighted the improved efficiency of recovery in QUIC as one of the benefits.
I recall discussing something of sort in and , but I'm not sure if we considered these side effects enough. IIRC, we abandoned the discussion to replace Ignore Order bit with Reordering Threshold, partly because we could not find a design that is easy to implement. But now, I think I might have a solution. We could: Replace the Ignore Order field with a Reordering Threshold field. State that receiver SHOULD ack every 2 packets that it receives, whenever the number of packets contained in the highest ACK range is no greater than that threshold. A sender that uses a threshold of 3 can advertise Reordering Threshold of 3 for example, to expect an immediate ACK to be sent when the receiver receives 3 packets after a gap. Or, a sender can advertise 5 instead, so that the loss would be noticed at an early moment even when there is an ACK loss. WDYT?
This is exactly what I was trying to mitigate with , but I think you explained the issue better than I did. With the current algorithm, QUIC would end up falling back to the time threshold loss detection, opposed to packet threshold, so it wouldn't be that bad, but there could be a delay. I'd like to agree on the properties we want. I think we want to ensure either time threshold or packet threshold are able to be triggered as quickly as possible. Given that, I think we need to send one (at least relatively, see last paragraph) immediate ACK upon observing out of order packets. That ensures that if only one packet arrives after the missing packet, it is detected in a timely manner by time threshold. Second, I think we want to avoid sending any other ACKs except the one which will trigger packet threshold. So the slightly complex bit is getting the algorithm for the second part correct. If you agree with the above goals, I think we need something a bit more complex than what you proposed. ie: What if the reordering threshold is 5 and you receive LostPN + 1, then LostPN+3, so there are already multiple ranges between the packet that could be declared lost and the reordering threshold. I'll try to describe and algorithm for that second ACK: Track the largestacknowledgedsent in an ACK When there are missing packets in the range , send an immediate ACK. Additionally, I think we should point out that small scale reordering(ie: 1 packet or twiddles) are the most common, so it makes sense to process an entire batch of available packets before determining whether to send an immediate acknowledgement.
I see, sorry for revisiting the issue. I hope we'd resolve the issue correctly this time. Makes sense. The properties that I had in my mind were slightly different from what you have: send ACKs no more than vanilla QUIC v1 send ACKs to make sure that losses are detected as early as vanilla QUIC v1 provide opportunity to deal with ACK-losses, so that even when there are ACK losses, losses can be detected as early as vanilla QUIC v1 The reason I would not mind sending two or three ACKs to indicate one gap is because packet losses are rare. Let’s say that average packet loss ratio is 1%. Then, assuming Reordering Threshold of 3, the debate is between if the number of ACKs being sent in response to gaps would be ~1% of the packets that the sender sends, or ~2% of the packets that the sender sends (in case of the algorithm described in URL). I must admit that the number of ACKs used for indicating one gap increases relative to the Reordering Threshold, but I’m not sure if we want to use values above 3. All that said, I would be fine with the properties you describe, as I think the required algorithm is going to be simple enough, and because ACK loss is not a big problem assuming that packet losses are rare in general. Please Ignore my comments below this line, I see the algorithm being corrected in the comment above :+1: Though I think there's a tiny bug in the algorithm being proposed: I think this should be something like: When the new largest acknowledged is : , send an immediate ACK. Otherwise, if an ACK is sent due to other reasons immediately after a gap, then that gap would not be notified at the earliest moment. One example is PN=255 being lost, then PN=256 received and ACK being sent due to Ack-Eliciting Threshold. When the receiver receives PN=257 and PN=258, then it should send an immediate ACK, assuming Reordering Threshold of 3.
Thanks, I just updated my suggested algorithm, and I think it now matches yours, but is stated differently? We use adaptive reordering, so it's not uncommon for our threshold to be 10+, which is 5 extra ACKs. Also, the every 2 option still requires the time threshold detection if you get one too few packet, whereas you could send an ACK at exactly the right threshold and the peer would immediately declare a loss. Not a big deal, but it's slightly suboptimal.
+1 to everything NAME says right above.
As I said on the mike during the WG meeting, there are two ways of distinguishing expected renumbering from packet losses. One is to check sequence numbers, and assume packet loss if >N packets received after hole. Another is a to check a reordering delay, if >T passes and the hole is still there. What I found in practice is that reordering delay thresholds work better than reordering number thresholds. Would be nice if the PR allowed for specifying a reordering delay, independent of the already specified ACK delay.
So let me understand the goal: you want to force an acknowledgment if a gap persists. A gap appears because a packet with a larger packet number is received. Then: Time: At that time, start a timer for some time less than the maximum acknowledgment delay and send an acknowledgment if the packet is still missing when the timer pops. (I'm not sure how to do this without having a timer running for every missing packet.) Number: This only requires that N packets after the gap arrive. I think that I understand Ian's algorithm for this.
In practice, you check the time of arrival of the following packets, and if the gap-delay is exceeded you fire the ACK.
NAME I think the point here is that the receiver has to remember when each packet was received, in addition to the packet numbers. That means that QUIC stacks have to change to the design of their ack queue, which makes this extension something much harder to implement than what it is now. At the moment, this extennion does not require changes to existing data structure. Separately, I am not sure how valuable it is to specify a signalling mechanism for time-based gap detection. ACKFREQUENCY frame already has Request Max Ack Delay field for time-based signalling. Senders can set this value to a fraction of RTT (e.g., 9/8 RTT) to induce ACK earlier than when the senders' time-based loss detection should be kicked. Such approach does have the downside of Inducing ACKs regardless of having gaps. But IMO, that is not a big issue, because: Most congestion controllers require ACKs to be sent within a fraction of RTT. Reducing the number of ACKs from one per two ack-eliciting packets to one per fraction of RTT is important (because the reduction of ACKs by their numbers is huge), but going below that is not important. Considering these aspects, I am personally happy with only_ having the packet-number based gap reporting mechanism as proposed by Ian in .
NAME I suppose that there is some equivalence between "wait for N packets" and "wait for T milliseconds". If I had to not use a time delay, I would set N = CWIN*T/RTT. Or vice versa. After all, the scope of the reordering is either a fraction of the CWIN or a fraction of the RTT, depending how you look at it. I get the feedback that reasoning on packet numbers is easier for several implementations, so OK. Let's just use packet numbers.
I've looked at the pull request and I think that the technical mechanism is sound. However, it took me a long time to get there using this discussion and the proposed text. The explanation needs a lot of work. I'll break this into several comments. The definition NAME uses is correct, but it is very much non-obvious from the explanation. More text is needed in the pull request in order to explain why this is the case. The goal here is to ensure that a new acknowledgment is sent that includes a (new) largest acknowledged that is more than the reordering threshold past a gap. An acknowledgment like that will cause the packets in that gap to be declared lost. That is why you look for a gap before (note: [^1]). However, you don't want to send an immediate ACK if the other side already has actionable information. If the gap is before , there is no point in sending an ACK because the sender will have already declared those packets lost. All of that needs to be clearly articulated. I would also call attention to a critical difference between this and the ack-eliciting threshold. The ack-eliciting threshold counts unacknowledged packets, without paying any attention to their packet number. If there is a massive gap, that won't care. The reordering threshold does care about packet numbers. That leads to the point that the two values are not related in any way. Either one can be larger with good reason, depending on how the sender is operating. Now we get to the harder bits of feedback... This change highlights how there are two piece of logic in play: The logic that determines when to send an immediate ACK in the presence of apparent reordering. In , this occurs when any gap appears. That equates to a recommended reordering threshold here of 1 (I think). The logic that determines when a packet is lost when processing ACK frames. In kPacketThreshold = 3. This document sets both to the same value and provides no way to separate the two. I think that's OK, but it is worth noting. As a matter of principle it seems tolerable for the former to be smaller than the latter, as we did in the original RFCs, but it is worth aligning the two on the basis that getting out of sync can lead to immediate acknowledgments that don't directly help with loss recovery. Again, I think that it is worth discussing this change, so that it is clear that the change is deliberate and to help readers understand what is going on. I can't see any functional difference between a reordering threshold of 0 and 1. Is that right? Could we benefit from changing the encoding to eliminate this redundancy? (I don't think we need to here - neither value is nonsensical - but I wanted to double-check.) Finally, I think that some examples would do a lot to improve comprehension. [^1]: Please don't call this , it's confusing: the logic here is going to decide whether this becomes the largest acknowledged packet; until then, it is the largest received packet (number).
NAME I think your point about the two different values should be added. And the logic in PR also needs to be clarified. But maybe we can just merge PR for now and then start another PR?
I think some wording clarifications would improve the text.

Discussion:
intensive, and reducing acknowledgement frequency reduces this cost at a data sender. <del> Severely asymmetric link technologies, such as DOCSIS, LTE, and satellite links, connection throughput in the data direction becomes constrained when the reverse bandwidth is filled by </del> <ins> For severely asymmetric link technologies, such as DOCSIS, LTE, and satellite links, connection throughput in the forward path can become constrained when the reverse path is filled by </ins> acknowledgment packets.  When traversing such links, reducing the <del> number of acknowledgments allows connection throughput to scale much further. </del> <ins> number of acknowledgments can achieve higher connection throughput. The rate of acknowledgment packets can impact link efficiency, including transmission opportunities or battery life. </ins> As discussed in implementation however, there can be undesirable consequences to congestion control and loss recovery if a receiver

QUIC-TRANSPORT currently specifies a simple delayed acknowledgement mechanism that a receiver can use: send an acknowledgement for every <del> other packet, and for every packet when reordering is observed.  This simple mechanism does not allow a sender to signal its constraints. This extension provides a mechanism to solve this problem. </del> <ins> other packet, and for every packet that is received out of order (Section 13.2.1 of QUIC-TRANSPORT).  This simple mechanism does not allow a sender to signal its constraints.  This extension provides a mechanism to solve this problem. </ins> 3.

A variable-length integer representing the maximum number of ack- eliciting packets the recipient of this frame can receive without <del> sending an immediate acknowledgment.  An immediate acknowledgement is sent when more than this number of ack-eliciting packets have been received, so value of 0 results in an immediate acknowledgement.  If an endpoint receives an ACK-Eliciting Threshold value that is larger than the maximum value the endpoint can represent, the endpoint MUST use the largest representable value instead. </del> <ins> sending an acknowledgment.  In other words, an acknowledgement is sent when more than this number of ack-eliciting packets have been received.  Since this is a maximum value, a receiver can send an acknowledgement earlier.  A value of 0 results in a receiver immediately acknowledging every ack-eliciting packet. </ins> A variable-length integer representing the value to which the endpoint requests the peer update its "max_ack_delay"

which is in milliseconds.  Sending a value smaller than the "min_ack_delay" advertised by the peer is invalid.  Receipt of an invalid value MUST be treated as a connection error of type <del> PROTOCOL_VIOLATION. This field has no meaning in this version of ACK_FREQUENCY.  The value of this field MUST be 0x00.  Receipt of any other value MUST be treated as a connection error of type FRAME_ENCODING_ERROR. </del> <ins> PROTOCOL_VIOLATION.  Reordering Threshold: </ins> <del> A 1-bit field representing a boolean truth value.  This field is set to "true" by an endpoint that does not wish to receive an immediate acknowledgement when the peer observes reordering (reordering).  0 represents 'false' and 1 represents 'true'. A 1-bit field representing a boolean truth value.  This field is set to "true" by an endpoint that does not wish to receive an immediate acknowledgement when the peer receives CE-marked packets (reordering).  0 represents 'false' and 1 represents 'true'. </del> <ins> A variable-length integer that indicates how many out of order packets can arrive before eliciting an immediate ACK.  If no ACK_FREQUENCY frames have been received, this value defaults to 3, which is the recommended packet threhold for loss detection in (Section 18.2 of QUIC-RECOVERY).  A value of 0 indicates immediate ACKs SHOULD never be sent due to receiving an out-of-order packet. </ins> ACK_FREQUENCY frames are ack-eliciting.  However, their loss does not require retransmission if an ACK_FREQUENCY frame with a larger

6. <del> The IMMEDIATE_ACK Frame is a frame which causes the peer to send a packet containing an ACK frame immediately, similar to the receipt of Initial and Handshake packets during the QUIC handshake. </del> <ins> A sender can use an ACK_FREQUENCY frame to reduce the number of acknowledgements sent by a receiver, but doing so increases the chances that time-sensitive feedback is delayed as well.  For example, as described in loss, delaying acknowledgements can increase the time it takes for a sender to detect packet loss.  The IMMEDIATE_ACK frame helps mitigate this problem. An IMMEDIATE_ACK frame can be useful in other situations as well. For example, it can be used with a PING frame (Section 19.2 of QUIC- TRANSPORT) if a sender wants an immediate RTT measurement or if a sender wants to establish receiver liveness as quickly as possible. </ins> <del> Receivers of the IMMEDIATE_ACK frame MAY choose to delay sending the ACK if the vast majority of received packets contain an IMMEDIATE_ACK or the receiver is under heavy load.  Senders MAY include multiple IMMEDIATE_ACK frames in a single QUIC packet, but the behavior is identical to a single IMMEDIATE_ACK frame. </del> <ins> An endpoint SHOULD send a packet containing an ACK frame immediately upon receiving an IMMEDIATE_ACK frame.  An endpoint MAY delay sending an ACK frame despite receiving an IMMEDIATE_ACK frame.  For example, an endpoint might do this if a large number of received packets contain an IMMEDIATE_ACK or if the endpoint is under heavy load. </ins> 7.

Since the last acknowledgement was sent, "max_ack_delay" amount of time has passed. <del> reordering, congestion, and batch describe exceptions to this </del> <ins> out-of-order, congestion, and batch describe exceptions to this </ins> strategy. An endpoint is expected to bundle acknowledgements when possible.

reordered ack-eliciting packet.  This extension modifies this behavior. <del> If the endpoint has not yet received an ACK_FREQUENCY frame, or if the most recent frame received from the peer has an "Ignore Order" value of "false" (0x00), the endpoint MUST immediately acknowledge any subsequent packets that are received out of order. If the most recent ACK_FREQUENCY frame received from the peer has an "Ignore Order" value of "true" (0x01), the endpoint does not make this exception.  That is, the endpoint MUST NOT send an immediate </del> <ins> If an endpoint has not yet received an ACK_FREQUENCY frame, the endpoint immediately acknowledges any subsequent packets that are received out of order, as specified in Section 13.2 of QUIC- TRANSPORT.  An endpoint, that receives an ACK_FREQUENCY frame with a Reordering Threshold value other than 0x00, MUST immediately send an ACK frame when the packet number of largest unacknowledged packet since the last detected reordering event exceeds the Reordering Threshold. If the most recent ACK_FREQUENCY frame received from the peer has a "Reordering Threshold" value of 0x00, the endpoint does not make this exception.  That is, the endpoint MUST NOT send an immediate </ins> acknowledgement in response to packets received out of order, and instead continues to use the peer's "Ack-Eliciting Threshold" and "max_ack_delay" thresholds for sending acknowledgements.

While it is expected that endpoints will have only one ACK_FREQUENCY frame in flight at any given time, this extension does not prohibit <del> having more than one in flight.  Generally, when using "max_ack_delay" for PTO computations, endpoints MUST use the maximum of the current value and all those in flight. </del> <ins> having more than one in flight.  When using "max_ack_delay" for PTO computations, endpoints MUST use the maximum of the current value and all those in flight. </ins> When the number of in-flight ack-eliciting packets is larger than the ACK-Eliciting Threshold, an endpoint can expect that the peer will not need to wait for its "max_ack_delay" period before sending an acknowledgement.  In such cases, the endpoint MAY therefore exclude <del> the peer's 'max_ack_delay' from its PTO calculation.  Note that this optimization requires some care in implementation, since it can cause premature PTOs under packet loss when "ignore_order" is enabled. </del> <ins> the peer's 'max_ack_delay' from its PTO calculation.  When Ignore Order is enabled and loss causes the peer to not receive enough packets to trigger an immediate acknowledgement, the receiver will wait 'max_ack_delay', increasing the chances of a premature PTO. Therefore, if Ignore Order is enabled, the PTO MUST be larger than the peer's 'max_ack_delay'. </ins> 9. <del> There are tradeoffs inherent in a sender sending an ACK_FREQUENCY frame to the receiver.  As such it is recommended that implementers experiment with different strategies and find those which best suit their applications and congestion controllers.  There are, however, noteworthy considerations when devising strategies for sending ACK_FREQUENCY frames. </del> <ins> This section provides some guidance on a sender's choice of acknowledgment frequency and discusses some additional considerations.  Implementers can select an appropriate strategy to meet the needs of their applications and congestion controllers. </ins> 9.1. <del> A sender relies on receipt of acknowledgements to determine the amount of data in flight and to detect losses, e.g. when packets experience reordering, see QUIC-RECOVERY.  Consequently, how often a receiver sends acknowledgments determines how long it takes for losses to be detected at the sender. </del> <ins> A sender needs to be responsive to notifications of congestion, such as a packet loss or an ECN CE marking.  Also, window-based congestion controllers that strictly adhere to packet conservation, such as the one defined in QUIC-RECOVERY, rely on receipt of acknowledgments to send additional data into the network, and will suffer degraded performance if acknowledgments are delayed excessively. To enable a sender to respond to potential network congestion, a sender SHOULD cause a receiver to send an acknowledgement at least once per RTT if there are unacknowledged ack-eliciting packets in flight.  A sender can accomplish this by sending an IMMEDIATE_ACK frame once per round-trip time (RTT), or it can set the Ack-Eliciting Threshold and Request Max Ack Delay values to be no more than a congestion window and an estimated RTT, respectively. </ins> 9.2. <del> Many congestion control algorithms have a startup mechanism during the beginning phases of a connection.  It is typical that in this period the congestion controller will quickly increase the amount of data in the network until it is signalled to stop.  While the mechanism used to achieve this increase varies, acknowledgments by the peer are generally critical during this phase to drive the congestion controller's machinery.  A sender can send ACK_FREQUENCY frames while its congestion controller is in this state, ensuring that the receiver will send acknowledgments at a rate which is optimal for the the sender's congestion controller. </del> <ins> Receiving an acknowledgement can allow a sender to release new packets into the network.  If a sender is designed to rely on the timing of peer acknowledgments ("ACK clock"), delaying acknowledgments can cause undesirable bursts of data into the network.  A sender MUST limit such bursts.  In keeping with Section 7.7 of QUIC-RECOVERY, a sender can either employ pacing or cause a receiver to send an acknowledgement for at least each initial congestion window of received data. </ins> 9.3. <del> Congestion controllers that are purely window-based and strictly adherent to packet conservation, such as the one defined in QUIC- RECOVERY, rely on receipt of acknowledgments to move the congestion window forward and send additional data into the network.  Such controllers will suffer degraded performance if acknowledgments are delayed excessively.  Similarly, if these controllers rely on the timing of peer acknowledgments (an "ACK clock"), delaying acknowledgments will cause undesirable bursts of data into the network. </del> <ins> Acknowledgements are fundamental to reliability in QUIC. Consequently, delaying or reducing the frequency of acknowledgments can cause loss detection at the sender to be delayed. A QUIC sender detects loss using packet thresholds on receiving an acknowledgement (Section 6.1.1 of QUIC-RECOVERY); delaying the acknowledgement therefore delays this method of detecting losses. Reducing acknowledgement frequency reduces the number of RTT samples that a sender receives (Section 5 of QUIC-RECOVERY), making a sender's RTT estimate less responsive to changes in the path's RTT. As a result, any mechanisms that rely on an accurate RTT estimate, such as time-threshold loss detection (Section 6.1.2 of QUIC- RECOVERY) or Probe Timeout (Section 6.2 of QUIC-RECOVERY), will be less responsive to changes in the path's RTT, resulting in either delayed or unnecessary packet transmissions. To limit these consequences of reduced acknowledgement frequency, a sender SHOULD cause a receiver to send an acknowledgement at least once per RTT if there are unacknowledged ack-eliciting packets in flight.  A sender can accomplish this by sending an IMMEDIATE_ACK frame once per round-trip time (RTT), or it can set the Ack-Eliciting Threshold and Request Max Ack Delay values to be no more than a congestion window and an estimated RTT, respectively. A sender might use timers to detect loss of PMTUD probe packets.  A sender SHOULD bundle an IMMEDIATE_ACK frame with any PTMUD probes to avoid triggering such timers. </ins> 9.4. To avoid additional delays to connection migration confirmation when using this extension, a client can bundle an IMMEDIATE_ACK frame with the first non-probing frame (Section 9.2 of QUIC-TRANSPORT) it sends <del> or it can simply send an IMMEDIATE_ACK frame, which is a non-probing </del> <ins> or it can send only an IMMEDIATE_ACK frame, which is a non-probing </ins> frame. An endpoint's congestion controller and RTT estimator are reset upon

connection SHOULD update and send a new ACK_FREQUENCY frame immediately upon confirmation of connection migration. <del> 9.5. A sender might use timers to detect loss of PMTUD probe packets.  A sender SHOULD bundle an IMMEDIATE_ACK frame with any PTMUD probes to avoid triggering such timers. </del> 10. TBD.