Change:
An issue came up in discussions around WebTransport: URL The problem is that datagram frames may be acknowledged for the sake of congestion control, but dropped for the sake of flow control. A receiver with a full datagram buffer would have two choices: Drop any packets with datagram frames and do not acknowledge them. Drop the datagram frames but acknowledge the packets. The first approach seems terrible because it drop any other frames bundled with the datagram (ex. STREAM) and cause retransmissions. It would also be treated as network congestion by the sender, causing limited bandwidth for otherwise unrelated STREAM frames. The second approach has ramifications for any implementations that MAY expose datagram acknowledgements. Specifically, the sender would believe that the datagram has been received (although not necessary processed), when in reality it could have been dropped and will never be processed.
I suggest that we remove the recommendation that datagram acknowledgements can be exposed to the application, and possibly even prohibit it.
Alternatively, this problem could be avoided if there was explicit flow control for datagrams. This would avoid a full datagram buffer and may have other benefits.
Adding flow control to DATAGRAM seems like a problem for use cases that don't want that, like tunneling of UDP-based transports that provide their own flow control
I may be wrong, but this seems very implementation specific, but IMO flow control is unnecessary so long as datagrams are indicated to the application layer inline (no buffering). For instance, MsQuic only acknowledges a packet with a datagram frame if it's able to successfully process it (datagram frame processing ). Additionally, "processing" a datagram frame includes indicating it up (inline indication ) to the application layer. There is absolutely no buffering at the datagram frame layer in MsQuic; so flow control is not a really issue. It's possible the application layer then does some buffering and would require flow control, but that's completely up to the application layer. If there is a scenario where the application layer may drop data (due to buffering constraints or something else) it should be up to the application layer to decide if the peer needs to know that information; and if so, inform them appropriately at their layer.
Surely that could be addressed by specifying arbitrarily large or infinite flow control limits? Would still complicate the spec, which seems unfortunate. That's a really cool solution. It's not a small constraint on the receiver's implementation, but it's certainly a case where we forbidding application-visible ACKs would be a harmful restriction.
Inline datagram processing is a cool idea and removes the need for flow control. Unfortunately, it's the sender that exposes acknowledgements to the application, but it's the receiver that would need to process and acknowledge datagrams inline. The sender would not be able to assume the receiver uses a particular implementation.
The capability to do inline indications on the receive side could be negotiated with the transport parameter. "Hey, I receive datagrams, and I don't buffer them." Then the sender side can decide how they want to handle ACKs based on that state. Though, it's not great that they still have to handle the scenario of "I do buffer, and I might throw them away" so long as any peer they communicate with might buffer. Personally, I'd just like to say that we require guaranteed app delivery for acknowledged QUIC packets with datagram frames, and then let QUIC implementations sort it out, but I realize that's not easily done for most implementations without significant refactoring. But if we don't do this, I do see it as a major bummer that buffering on the receiver side could effectively make QUIC layer acknowledgements worthless to the app (which IMO was the best advantage to this solution).
My only other thought would be to have a new frame (e.g. DATAGRAM_DROPPED) that a receiver sends if it drops DATAGRAM frame; but to accurately identify which that were dropped, each datagram would need a unique identifier and that further complicates all this. So I definitely wouldn't prefer this solution.
Or, in more explicit terms, "My ACKs guarantee delivery of datagram frames to the application." Is this any different from handling the scenario of a peer that doesn't support datagram frames at all?
I agree with NAME earlier comment that this sounds very implementation-specific. Our implementation also would not drop any datagram frames that have been received and ack'ed, since we process the data immediately and send it inline. I'd almost prefer to just have a SHOULD NOT drop datagram frames if they've been received and ack'ed. This will certainly influence API design, and means that you'll have a rougher time if you just try to imitate a UDP socket buffer.
Yes, there is a difference. If the peer doesn't support datagram frames at all, the protocol might not work at all, and the app layer will just reject the connection outright. If the app supports datagram frames, but it's implementation may or may not buffer. In that case, depending on the app protocol, it may or may not then need additional app layer acknowledgements or accounting. For instance, perhaps some game protocol that has a fire-and-forget "current player state" datagram payload could care less about acknowledgements, so it doesn't care if buffering drops a frame. All it needs is datagram support.
I would also prefer this approach.
For an implementation that doesn't expose the contents of QUIC packets to an application, exposing the ACKs has little value. The flow control issue is then a little moot. I would push back on any strict requirements as mentioned, it presupposes that implementations will provide a single style of processing or API. SHOULD NOT seems acceptable because I would drop them based on some criteria and the text probably encourages me to document this aspect.
In the context of WebTransport, datagrams are processed at the Javascript level. Browsers are understandably not going to execute and block on inline Javascript callbacks for every received datagram.
Why can't the app layer reject the connection outright if the peer's transport parameters indicate that it does not support reliable acknowledgement for datagrams, just as it would if they indicate no support for datagrams at all? An app which doesn't care about ACKs obviously wouldn't do so, but those that do, can. Or they could apply app-layer mitigations, but that's more options, not fewer.
+1 to URL . There is a process hop between the QUIC code and the JavaScript code, and the QUIC code needs to send ACKs quickly (especially if that packet contains a STREAM frame next to a DATAGRAM frame) so it's simply not possible to wait on JavaScript before sending the ACK. You could buffer datagrams indefinitely after ACking them to ensure that the application gets them, but that goes against the entire philosophy of DATAGRAMs by increasing latency unboundedly. "SHOULD NOT drop DATAGRAM frames" sounds completely unrealistic to me.
NAME You don't need to expose how a datagram frame buffer from the app is put in a QUIC packet in order to give them information back about the receipt (or not) of an acknowledgement of their data. MsQuic allows the app to give a context pointer along with the buffer they send. We then indicate acknowledgement or suspected loss information back to the app for that context (event ). NAME and NAME I totally agree that it's unreasonable to expect every layer above QUIC in the WebTransport scenario to do inline delivery, but that isn't the only scenario, and having that one scenario's requirement for buffering shouldn't make us throw our hands up and say no scenarios will be able to use a zero buffering approach. Protocols directly on top of QUIC should be able to do whatever they want. The "SHOULD NOT drop DATAGRAM frames" purely at the QUIC layer is not unrealistic IMO. What the layers on top of QUIC do are totally up to them to define and implement. WebTransport datagrams processed at the Javascript layer will need application layer acknowledgements if the protocol requires info on exactly which datagrams were delivered.
Thanks Nick, that makes more sense. I do wonder though how useful it acks are if you're talking to my endpoint who might, in some conditions, drop the occasional DATAGRAM on the floor.
NAME I was thinking of this in terms of WebTransport since that was really top of mind. I agree that different applications have different needs, and some might be able to do this inline ACKing more than others. That said, I'm really not sure what the point of this SHOULD NOT is, it's like saying "an IP router SHOULD NOT drop IP packets" - most router implementors know that they shouldn't drop packets unless they have a good reason such as having nowhere to put the packet.
NAME I do agree somewhat with that sentiment, but it's a bit different because the transport is saying "Yes, I got the packet and was able to successfully process it.". If it's possible/allowable for the transport to then just drop it on the floor, is there really any reason to acknowledge datagram frames at all, beyond just another RTT estimate? I just feel it would be a big shame that we have all this ACK machinery built into QUIC, but it gets wasted on datagrams. Not only that, it has to be rebuilt completely by the app to duplicate all the work QUIC already does.
NAME the way I see it, there are only two ways to perform queueing when you have one input and one output: option 1: you buffer things indefinitely forever option 2: you don't I really don't see any other way. I think you're suggesting that transports should buffer DATAGRAMs infinitely if the application is busy, and that doesn't sound like a good idea to me.
There are two questions here: who owns the buffers -- QUIC, or the application; and what are the application's reliability requirements. I would argue that for maximizing generality of a DATAGRAM receiver implementation -- so that multiple application behaviors are possible -- you want the application to own these buffers. Whether these buffers are implemented in QUIC or at the application or somewhere in between doesn't matter; it is meant for the application. For an application that wants some notion of reliability, there are two choices in how a receiver manages its buffers: it signals delivery out of the buffers and/or loss (ACKs/NACKs/gaps); it signals buffer size explicitly to avoid any loss (flow control) An app that sits on top of QUIC DATAGRAMs thus has two choices: use explicit Application ACKs to signal receipt (and gaps or NACKs to signal loss), since the sender might have caused the application buffer to overflow; use explicit flow control to disallow application buffer overflowing. Doing this has the nice property that QUIC ACKs can then be treated as application ACKs, since application flow control will ensure that this data does not get dropped at the receiver between QUIC and the app. In either case, it's a decision that is both application- and implementation-dependent. An app that cannot know the buffer size will have to rely on using explicit application ACKs. An app that is ok with data loss can allow loss to happen, and may or may not signal this back. An app that wants to avoid buffer overflow at the application buffers can reuse QUIC ACKs as application ACKs but will need to ensure that application flow control is in place. Implementations ought to expose DATAGRAM QUIC ACKs to the application, so that applications have a choice of using that signal. However, the application is ultimately responsible for using them correctly.
NAME the way I read your summary of the issue (which is great, btw. Thanks!) it seems that not only do ACKs need to be exposed to the application, but a receiver can't drop acknowledged datagrams without informing the application, right? Otherwise, how would the app be able to reliably implement it's own flow control along with QUIC ACKs? A sender who receives an ACK of a datagram would assume that datagram is taking up buffer on the peer application; but if QUIC dropped it before it received that buffer, then both application sides would get out of sync in their view of the FC window. The sender could eventually stall out because of this, right?
"Instant processing" model is really nice, and makes sense if you're in the same process as the QUIC stack, but across a privilege separation boundary (web app/browser, userspace/kernel, etc), this is not feasible, so I don't think this should be allowed at least by default. &nbsp;this is technically possible, and it is in fact how WebSocket API works; if your web app is too slow at processing incoming WebSocket messages, your server will just crash a browser tab remotely by overflowing the task queue, so most people agree this was a big mistake.
After reading through this thread I may have missed it, but has anyone considered simply having the spec say something along the lines of: "If an ACK frame has been sent and then due to implementation specific buffering or resource constraints it is unable to process the corresponding DATAGRAM frame, the application MUST be notified of the locally dropped frames so that it has the option to support additional signaling if necessary." This will ensure that only for any implementation where this case is possible, the application developer/operator will be made aware of such a possibility and can decide how best to handle it either through configuration or custom application logic.
What could a reasonable application do with that notification, aside from assert that it doesn't happen?
Many things are possible with the notification, the ability to recognize when the dropped packets are not due to network congestion could be used to do custom throughput signaling back to the sender that incorporates other local signals such as queue depth or processing rate. The underlying implementation might also expose dynamic controls on the cache or queue size that the application can adjust. It could also back off other memory or processor heavy operations in order to prioritize capacity to not drop future packets. By ensuring that the ACK signaling only indicates congestion it isolates the responsibility of additional flow control signaling to just implementations that support it, only having to support notifying applications in such an event where frames are dropped locally.
Dropping datagrams ought to be similar to resetting a stream. It is pretty much the same semantic.
NAME can you elaborate? Resetting a stream is a non-recoverable situation for a stream, whereas dropping datagrams is recoverable.
Resetting streams is defined as an application triggered message: application says it cannot proceed with the stream and provides an application defined code and reason. I think the application saying "I am dropping some of the datagrams that I received" has similar semantics.
I see. It would make me sad to turn a recoverable error into an unrecoverable one though, because some applications may not want that.
Yes, that's where the analogy with reset breaks. The datagram dropped message should be informative, not change the state of the connection or the datagram streams.
The difficulty I have with this concept is that I don't know what I'd expose to the application. My datagram implementation basically keeps a fixed-size queue of frames. So if I dropped some all I can tell an application is that something got dropped. Adding more information about what was dropped increases complexity. The library already doesn't expose information about packets to the application, and so far things have worked out OK.
I tend to agree that there isn't much a QUIC stack that queues datagrams can do here. The best it could do it indicate how many it dropped, but that's likely not enough. If that data needs to be retransmitted then (at the app layer) they would likely need more detailed info (such as an app identifier for each dropped packet). I think the best the draft can do here is mention that if the QUIC stack drops datagrams frames and also acknowledges them, then those acks cannot be leveraged by the application layer. It might be nice to have an additional TP that allows behavior this to be negotiated, but it's not strictly necessary.
" If that data needs to be retransmitted then (at the app layer) " ... then the app needs to run a transport on top of datagrams. Or use regular streams.
+1, if you need reliability then datagrams are not the droids you're looking for
The more I think on this, the more it just seems similar to the existing notion that we have with QUIC streams; transport delivery != application end-to-end delivery & processing. It's like email read-reciepts: just because I opened the email doesn't mean I read it, digested it and have a response. The doc might benefit from doing a better job of explaining this notion and the interop concerns that might arise. For instance, stating that it is possible for receiving implementations to acknowledge DATAGRAM-bearing packets while providing no guarantee that the DATAGRAMS are presented to the application. Valid reasons are flow control or any other arbitrary reason the receiver decides. Application's that desire end-to-end loss-detection need to be aware of this possibility and build measures that accomodate it. As Jana points out, there are some signals an implementation MAY chose to provide to make that easier. If they don't, applications can vote with their feet.
Adding text to clarify that receipt of an ACK for a packet containing a DATAGRAM frame does not mean that the application got the frame sounds like the right solution here.
Text is good, but I am still a bit sad that we don't have some way to leverage the QUIC ACKs in a general sense (not implementation specific).

Discussion:
If a sender detects that a packet containing a specific DATAGRAM frame might have been lost, the implementation MAY notify the <del> application that it believes the datagram was lost.  Similarly, if a packet containing a DATAGRAM frame is acknowledged, the implementation MAY notify the application that the datagram was successfully transmitted and received.  Note that, due to reordering, a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged. </del> <ins> application that it believes the datagram was lost. Similarly, if a packet containing a DATAGRAM frame is acknowledged, the implementation MAY notify the sender application that the datagram was successfully transmitted and received.  Due to reordering, this can include a DATAGRAM frame that was thought to be lost, but which at a later point was received and acknowledged.  It is important to note that acknowledgement of a DATAGRAM frame only indicates that the transport-layer handling on the receiver processed the frame, and does not guarantee that the application on the receiver successfully processed the data.  Thus, this signal SHOULD NOT replace application-layer signals that indicate successful processing. </ins> 5.3.