Code change:
I don't fully understand when would someone want to set ignore-order = true, i.e., what is a valid use case for this? Lets say for whatever reason the sender sets this to True, this would if there is actual packet loss and receiver receives OOO packets, it won't send an immediate ACK. Meaning the sender won't enter loss recovery immediately and will continue sending packets at the same rate. This is not good, as it directly increases queuing on the network node and for a tail drop queue (most widespread deployment), it will also affect other flows. Meaning, other flows will experience packet loss due to non-responsiveness of this flow for certain period of time. I probably missed past conversations about ignore-order field, so my apologies if this has already been discussed. I read the draft and didn't see any explanation about where would this be useful and what are the fallbacks of setting this to true.
One use case that I know and care is for sending QUIC traffic from a server cluster. Some packets would be sent directly from a server node that governs the connections, but others would be sent via different nodes within the same cluster. Once we deploy a sender like this, the receiver would observe what looks like reordering (because most packets will be essentially delivered through an additional hop, and that would cause delays). But we do not want the receiver to send ACK for each packet being received. Regarding the problem, I'm not sure I agree with the impact. Typically, congestion controllers react to congestion after 1 RTT. When packet-number-based detection is turned off, that changes to 9/8 RTT. That does cause more pressure on the bottleneck queue, but it is no worse than that caused by a path with an 9/8 greater RTT. Assuming that congestion controllers are designed to be fair against flows having different RTTs, I tend to believe that this is a non-issue.
I agree with NAME though I think adding a SHOULD NOT use a delay longer than an RTT(URL) would be a good addition, since if the delay is extremely large(ie multiple RTTs), that could be bad.
In general, the spirit of this draft is to provide control over acking behavior to the sender, since it is the consumer of this information and ought to be able to control how frequently it needs to see this information. If there are subtle consequences to these decisions, it is worth noting them in the draft. It seems useful to add some text on the consequences of using Ignore Order, so that senders can make informed decisions. We can do that.
NAME is this something that Fastly does or is planning to do? How would you share the connection secrets between different server nodes? When we set ignore-reorder = true, and lets assume that max ACK delay > 1RTT, it can be 1RTT or 2 or 10RTT. this would result in loss recovery to be delayed by at least an additional RTT, i.e. it will take >16/8RTT (>2RTT) instead of 9/8RTT to declare packets lost. That is significant IMO. >Assuming that congestion controllers are designed to be fair against flows having different RTTs, I tend to believe that this is a non-issue. CUBIC throughput is inversely proportional to RTT in New Reno mode. Nonetheless, I am more concerned about high queuing in bottleneck due to late loss recovery. Specifying the use-cases where ignore-order = true SHOULD be used will prevent new implementors from using it randomly. And as Jana said, some text around the consequences when it is used would allow informed decision making.
NAME I do not speak for Fastly's plan, but the OSS branch of H2O has a working prototype that does this: URL The calculation is correct. At the same time, I do not think that senders will be willing to use Max Ack Delay as large as that, when setting the Ignore Order bit. The problem here is the impact on loss recovery. If Max Ack Delay is set to a value as large as 1 RTT, then the sender needs to spend more than 2 RTTs for detecting a loss. That means that the size of the send and receive buffers have to be twice as large, and that it would take as twice as long to recover from a loss. Based on existing advice that we have in other RFCs, I think we should discourage Max Ack Delay that is greater than 9/8 RTT when Ignore Order bit is used (please see my comment on ).
I'd suggest 5/4, but otherwise agree with you NAME
Should the Max ACK Delay be restricted to 1/8 or 1/4 RTT as there is inherent 1RTT time spent while sending data and receiving ACKs? That would make a total of 1 + 1/8 or 1 +1/4 RTT total/
I think this is best as a sender-side SHOULD, since it knows what time threshold it's using. For example, we use 1/4 RTT, but have considered making it adaptive(ie: 1/16 RTT to 1 RTT) based on observed reordering. Also, if your RTT is 100us, and your timers have coarser granularity, 1/8 RTT or 1/4 RTT just doesn't make sense. I'm not sure what does, because I'm not an expert in datacenter networking, but that's why I think this is a SHOULD, not a MUST. Of course, it's also unenforceable, which is a good reason for a SHOULD.
+1 to SHOULD. The only argument I might disagree with NAME is that it is unenforceable (IMO a receiver can enforce by capping Ack Delay to a fraction of the RTT that the receiver has observed). But I do not think there's any reason to require the receiver to enforce, considering the fact that a sender can send as many packets as it likes, regardless of ACKs the receiver sends (or lack of).
I am fine with SHOULD and with a mention of how a sender would compute Max ACK Delay when he sets ignore-order = true. An example is fine too.
I believe this is the same issue as
There are two things that are impacted by setting a high value for ack delay: congestion control and time-threshold loss detection. On congestion control, we need some protection for the network, so we should recommend something. On loss detection, a sender that chooses and sets an ack delay higher than the time-threshold loss detection time period risks triggering unnecessary retransmissions. So, I propose adding the following for a sender: SHOULD set the ack frequency to be at least one ack per RTT (so that congestion control responses are reasonably quick) a word of caution that setting the ack delay to be longer than time threshold can cause the sender to unnecessary retransmit
In 9.3. Window-based Congestion Controllers I thought that unless there is loss reported, a QUIC ACK Frame releases sending window. In a similar manner as the accurate byte counting style in TCP, a QUIC sender solely operates on the basis of bytes acked, not the number of ACK frames received. So while a delayed ACK could delay a round of growth when the ACK Ratio is larger, it is only delayed by the time to ACK the set of received packets. In looking at various QUIC CC over medium and longer path RTTs this effect was quite small, and the default ACK delay was not unreasonable; for shorter path RTTs this might be different
See PR
To me, ACK Clocking is a separate point that applies to any type of CC when the cwnd limits the sender. I agree there could be a concern is around whether a less frequent ACK policy can induce a cwnd-limited sender to send bursts of packets that could induce loss or disrupt sharing of the path with other flows. The QUIC specification already permits an initial window of 10 packets, and motivates the need for pacing. I think it is important this text refers to the QUIC transport section on pacing to mitigate bursts.
Better worded perhaps in PR .
Section 9.1 - There are CC considerations with respect to how long it is reasonable for a flow to hold-off detecting congestion and responding. This needs to be discussed. When an endpoint detects persistent congestion, it MUST promptly reduce the rate of transmission when it receive or detects an indication of congestion (e.g., loss or ECN marking) [RFC2914], the Ignore Order value of true (0x01) in this ID allows a sender to extend that period, postponing detection of loss. That might be reasonable, but excessive delay can be dangerous - and therefore the impact really needs to be discussed: delaying a fraction of a RTT is in my mind safe, intentionally delaying by an RTT is arguable. A delay of many RTTs is endangers other flows, and we need to at least say that in some way ([RFC8084] took an alternate view of how long might be safe).
See PR
I think this warrants a note about how a sender SHOULD NOT use a delay that is larger than an RTT unless the sender has other information about either the network or current network conditions. Otherwise, the receiver does not need to enforce anything here.
NAME Delay of loss detection has negative impact on loss recovery as well. Considering that the time threshold defined in RFC 9002 is 9/8 RTT, I might ague that, when Ignore Order is set to true, then the maximum delay being advertised SHOULD be no greater than 9/8 RTT. These values provide responsiveness comparable to Rack when a packet is lost - the loss would be detected within 5/4 RTT. (stuck out, as this point is already covered by Section 8).

Related discussion:
text: 9. <del> There are tradeoffs inherent in a sender sending an ACK_FREQUENCY frame to the receiver.  As such it is recommended that implementers experiment with different strategies and find those which best suit their applications and congestion controllers.  There are, however, noteworthy considerations when devising strategies for sending ACK_FREQUENCY frames. </del> <ins> This section provides some guidance on a sender's choice of acknowledgment frequency and discusses some additional considerations.  Implementers can select an appropriate strategy to meet the needs of their applications and congestion controllers. </ins> 9.1. <del> A sender relies on receipt of acknowledgements to determine the amount of data in flight and to detect losses, e.g. when packets experience reordering, see QUIC-RECOVERY.  Consequently, how often a receiver sends acknowledgments determines how long it takes for losses to be detected at the sender. </del> <ins> A sender needs to be responsive to notifications of congestion, such as a packet loss or an ECN CE marking.  Also, window-based congestion controllers that strictly adhere to packet conservation, such as the one defined in QUIC-RECOVERY, rely on receipt of acknowledgments to send additional data into the network, and will suffer degraded performance if acknowledgments are delayed excessively. To enable a sender to respond to potential network congestion, a sender SHOULD cause a receiver to send an acknowledgement at least once per RTT if there are unacknowledged ack-eliciting packets in flight.  A sender can accomplish this by sending an IMMEDIATE_ACK frame once per round-trip time (RTT), or it can set the Ack-Eliciting Threshold and Request Max Ack Delay values to be no more than a congestion window and an estimated RTT, respectively. </ins> 9.2. <del> Many congestion control algorithms have a startup mechanism during the beginning phases of a connection.  It is typical that in this period the congestion controller will quickly increase the amount of data in the network until it is signalled to stop.  While the mechanism used to achieve this increase varies, acknowledgments by the peer are generally critical during this phase to drive the congestion controller's machinery.  A sender can send ACK_FREQUENCY frames while its congestion controller is in this state, ensuring that the receiver will send acknowledgments at a rate which is optimal for the the sender's congestion controller. </del> <ins> Receiving an acknowledgement can allow a sender to release new packets into the network.  If a sender is designed to rely on the timing of peer acknowledgments ("ACK clock"), delaying acknowledgments can cause undesirable bursts of data into the network.  A sender MUST limit such bursts.  In keeping with Section 7.7 of QUIC-RECOVERY, a sender can either employ pacing or cause a receiver to send an acknowledgement for at least each initial congestion window of received data. </ins> 9.3. <del> Congestion controllers that are purely window-based and strictly adherent to packet conservation, such as the one defined in QUIC- RECOVERY, rely on receipt of acknowledgments to move the congestion window forward and send additional data into the network.  Such controllers will suffer degraded performance if acknowledgments are delayed excessively.  Similarly, if these controllers rely on the timing of peer acknowledgments (an "ACK clock"), delaying acknowledgments will cause undesirable bursts of data into the network. </del> <ins> Acknowledgements are fundamental to reliability in QUIC. Consequently, delaying or reducing the frequency of acknowledgments can cause loss detection at the sender to be delayed. A QUIC sender detects loss using packet thresholds on receiving an acknowledgement (Section 6.1.1 of QUIC-RECOVERY); delaying the acknowledgement therefore delays this method of detecting losses. Reducing acknowledgement frequency reduces the number of RTT samples that a sender receives (Section 5 of QUIC-RECOVERY), making a sender's RTT estimate less responsive to changes in the path's RTT. As a result, any mechanisms that rely on an accurate RTT estimate, such as time-threshold loss detection (Section 6.1.2 of QUIC- RECOVERY) or Probe Timeout (Section 6.2 of QUIC-RECOVERY), will be less responsive to changes in the path's RTT, resulting in either delayed or unnecessary packet transmissions. To limit these consequences of reduced acknowledgement frequency, a sender SHOULD cause a receiver to send an acknowledgement at least once per RTT if there are unacknowledged ack-eliciting packets in flight.  A sender can accomplish this by sending an IMMEDIATE_ACK frame once per round-trip time (RTT), or it can set the Ack-Eliciting Threshold and Request Max Ack Delay values to be no more than a congestion window and an estimated RTT, respectively. A sender might use timers to detect loss of PMTUD probe packets.  A sender SHOULD bundle an IMMEDIATE_ACK frame with any PTMUD probes to avoid triggering such timers. </ins> 9.4.