Original text:
DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys. <del> Application protocols using datagrams are responsible for defining the semantics of the Datagram Data field, and how it is parsed.  If the application protocol supports the coexistence of multiple entities using datagrams inside a single QUIC connection, it can use a mechanism to allow demultiplexing between them.  For example, using datagrams with HTTP/3 involves prepending a flow identifier to all datagrams, see I-D.schinazi-quic-h3-datagram. </del> Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter, and by 5.1.  Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (I-D.ietf-quic-recovery).  Receivers SHOULD support delaying ACK frames (within the limits specified by a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged. <del> 5.2. </del>  DATAGRAM frames do not provide any explicit flow control signaling, and do not contribute to any per-flow or connection-wide data limit. frames are inherently unreliable, they MAY be dropped by the receiver if the receiver cannot process them. <del> 5.3. </del>  DATAGRAM frames employ the QUIC connection's congestion controller. As a result, a connection might be unable to send a DATAGRAM frame

Reviewer comments:
I would like this section to say something about priorities, as emerged in the issue. Something to the effect of: "QUIC Implementations SHOULD present an API to applications to assign relative priorities to DATAGRAM frames with respect to each other and to QUIC streams. This can be done by the API conceptually assigning DATAGRAM frames to an existing QUIC stream and inheriting its priority, or assigning a priority in the DATAGRAM send API itself."
the first sentence of NAME suggestion sounds ok. I don't like the second one much (sorry). edit: clarification - the first sentence is very much like RFC 9000's "A QUIC implementation SHOULD provide ways in which an application can indicate the relative priority of streams"
I'm not married to the second sentence.
Piling on to agree here: I'm OK with adding the first sentence, though I think the second one is too specific so I'd prefer the second sentence wasn't added
Added the first sentence about priorities. Please check!
I finally sat down and read this draft (nice work, BTW) and was somewhat sad to learn that QUIC won't natively support stream multiplexing for datagrams. It is true that applications can implement this if they choose. But one nice thing about QUIC is that it takes care of this machinery on behalf of the application. This is a somewhat aesthetic concern, but it feels awkward to have one layer of stream multiplexing in QUIC and then another layer in the application. It would be ugly in our implementation, at least. Moving on to non-aesthetic issues: I don't have a full grasp of all the use cases for DATAGRAM, but if stream multiplexing is a common requirement, it would be good to move this into the transport. Applications that don't need this are free to do it all over one stream at a tiny loss in wire efficiency, or we could have a stream-less version if people are deeply concerned about one byte per frame.
The argument for taking it out effectively identified that there is little point in putting something in the transport that the transport can't help with
While HOL blocking advantages don't apply here, I'm not sure that it's true that the transport couldn't help. For example, a stream construct would make it much easier to handle priority of DATAGRAMs in the send buffer. There are other ways to solve this, but with streams we could use existing QUIC priority APIs.
In Montreal I also argued for a design that could be used for streaming, but was out numbered at the time. NAME I agree with your opinion that it would be best for the transport to provide this kind of interface, rather than requiring every app/protocol to implement it on top of this extension. I think the best path forward would be make a separate draft for unreliable streams.
I guess I find this prioritization example confusing because the transport protocol doesn't include an expression of the priority on the wire. How does telling the remote endpoint the DATAGRAM flow ID (as we referred to it in previous discussions) help with things that are a concern for transport RX.
As with reliable streams, all the priority negotiation is at the application layer and indeed, whether the multiplexing is in the transport or application doesn't matter. However, in the reliable case it is certainly useful for the transport to have the notion of streams so that it can send the highest-priority streams first. It is certainly possible to build a QUIC API that accepts datagram priority in the absence of transport streams, but it seems a lot more convenient to reuse the existing semantics. If I'm in the rough here, this is not going to break the protocol. But it feels like a recipe for writing a bunch more code.
I do understand what you're saying and it sounds valid but I'm attempting to channel some counter arguments back here. One main services the transport provides for streams is accounting of data stream data flow windows and stream IDs. DATAGRAM is attractive to applications that don't require the reliability that streams provide and the lack of a flow ID means it also doesn't come with the overhead of accounting mechanisms that aren't so useful for single-use atomic messages. I do think DATAGRAM draft would benefit from some clarification about the expectations of prioritizing DATAGRAM data, especially against STREAMS. But I don't think the flow ID is a requirement for that.
A QUIC implementation could easily provide exactly the same API for managing datagram priority that it does for streams; neither need to be specified in detail by the draft, let alone present on the wire. I don't think adding a field to the wire representation could possibly save any code here; it's more complexity for every implementer, not less.
I agree with NAME point about not requiring every higher level protocol that needs it to separately implement flow IDs. The spec provides flow IDs. What if I am designing some new protocol to run on top of QUIC, other than HTTP/3, and I also have the need for multiple datagram flows? If the facility is in QUIC datagrams, then the QUIC library my protocol's implementors will be using is going to provide it to them for free. If the facility is in HTTP/3, few or no HTTP/3 implementations are going to allow just using bits of HTTP/3 in isolation from the rest of it. So it would force the implementors of my higher level protocol to reimplement this facility (or else I might just decide to base my protocol on top of HTTP/3 instead of QUIC – but that then gets away from the idea that QUIC is usable for more than just HTTP.) Couldn't the stuff about flow ID management in the HTTP/3 datagram spec just get moved to this spec instead?
One of the earlier drafts did have them combined exactly like that. The consensus of the group was to keep them separate. The main issue is that there is nothing at the transport layer that the flow ID influences, only the application. There’s nothing saying APIs can’t make this convenient to use, etc, but it doesn’t necessarily belong in the fundamental wire image of the datagram.
To satisfy the flow ID allocation requirement of H3-datagram all that is needed is something like How flow IDs are used is up to the application. And at that point, you're going to really benefit from well-formed application protocol and or/application-mapping. For example, there is no single answer to whether flow IDs are required nor how to bind flow IDs. So, IMO there really not a generic service that a QUIC library can provide to applications.
I personally disagree. The API for a library does more than just implement the protocol. It provides efficient ways to build application logic on top of protocol behavior too. Having the flow-ID at the protocol/library layer allows the API to provide a per-flow abstraction object, similar to a Stream object. This can allow for the app to associate different logic/context to different datagram flows (e.g. register a different receive callback for audio & video flows). IMO, this will be a very common usage on top of datagrams and will end up being reimplemented for every app/protocol built on top of datagrams. One of the primary points of the library/API to to implement logic once so that it may be reused for many different apps. Additionally, I don't believe this kind of abstraction layer should be in the library/API unless it's also in the protocol. The goal of the API is to be cross-compatible with other implementations, generically. If my implementation has some custom flow-ID logic, but no other API does, what's the likelihood of interop issues?
I could implement the same behaviour by registering a predicate to the QUIC library e.g. if N bytes of the QUIC DATAGRAM payload matches this pattern, visit this callback function. I don't have that today on the receive side but I do have that for purging pending items in my send queue, see URL I agree. If I added my predicate-style generic method to my QUIC library, I could build on top of it to support the H3-Datagram flow IDs as a first-class API property. It's the job of the application mapping implementation to write the correct predicate, and any additional logic that may be needed such as only accepting flows that have been permitted by some other form of negotiation (such as a CONNECT-UDP method). Just adding a flow ID field leaves open a lot of questions, and I don't think they can be answered commonly across all the potential applications.
From an API perspective, it is certainly useful to have the transport layer help demultiplex and put different flows in different send/receive queues in the application. To that end, I totally agree with you, NAME However, the diversity of use cases for tracking datagram flows, and the possible ways that this could be usefully exposed to the application, means that building in one interpretation of demultiplexing based on a flow ID without a use for the transport itself may be unnecessarily limiting. This is a common issue for how to do demultiplexing of application flows on top of a transport. In TAPS, we use protocol framers for the same thing, as a way for the application to tell the transport API "I want to separate messages based on this, and pull out these fields as special markers (like a flow ID)". (URL) I imagine that a QUIC API should follow this model if it wants to make it easy for datagram applications to demultiplex without having to build their own demux layers.
I confess to having a primarily aesthetic objection to having two different layers of stream multiplexing in the stack, but I've been trying to figure out a reason streams might matter for the transport. I certainly can't come up with a killer app for transport streams. The best I can find are purely send-side, so in theory they can be done with sufficiently rich APIs rather than something on the wire. I guess the question is whether or not this is too much load to put on the QUIC API: Priority. Without a transport-layer stream, the datagram API will have to specify priority on a per-datagram basis. Ordering. The draft should probably say a little more about ordering expectations, but obviously there are no firm guarantees here. I can certainly imagine QUIC reordering datagrams for its own reasons (e.g. sending a smaller packet first based on space available), and it might be good for the application to specify that a certain subset of datagrams SHOULD be delivered in order. To do this without transport-layer datagram streams, the API would have to provide some sort of datagram identifier that could be ordered with respect to other datagram identifiers. It sounds clunky, if this use case could conceivably matter to anyone. Of course, the higher-assurance way to do ordered delivery is to put a sequence number in the application datagram frame and discard OOO datagrams at the receiver. but it would be nice if QUIC didn't make things worse for these applications by arbitrary reordering. Martin
Let's not conflate "things it would be useful for a QUIC implementation to expose in its API" with "things that need to be mandated as part of the wire format." There can be discussion of options (not requirements) for the former in the text. Adding extra constraints to the wire format complicates all implementations, and makes applications with different requirements awkward.
Purely API-level stream identifiers would work fine for this, as I . This would be pretty convenient and I think is a sensible thing for the draft to suggest (but not require). The datagram extension as currently written is useful in part because it is so content-agnostic; there's plenty of room for more specialized extensions in the future if certain applications would benefit from a more complex wire format.
Yes, I agree that would solve the problem.
I like this status quo. I've been mulling whether in fact all streaming an multiplexing could be done top of a QUIC connection used just for datagrams without loss of QOS. It's certainly not the path taken so far, but I like this datagram extension being pretty minimal to keep the future wider open.
NAME I'm not sure that's the case here, the original design had an optional flow ID specified by the frame type, IIRC.
I tend to agree that this is something that almost every user of datagrams is going to need to specify, and being able to interoperate multiple different protocols using datagrams on a single QUIC connection has come up quite a bit. We handled this before by punting it to H3 and saying that you need H3 to be involved to get that mux-ing, but it's been noted in this thread that you might not always have H3 involved. Previously, this led us down into a conversation before we had figured out what alpn meant for QUIC, but at this point I think we've moved past that enough to not want each possible permutation of protocols in use/defined by a new alpn string to have a slightly different mechanism for handling flow identifiers, some of which might be compatible. If we think that introduces too much complexity/other problems to tackle, then right now we've just moved that complexity into H3, which means we (a) still need to solve it, but (b) now everyone else cannot take advantage of that solution.
To NAME TAPS comment, I think the question about having an additional [de]mux layer in between is the same as punting it to H3 -- we still need to figure out how to make that work, the question is just where do we group that effort and who can take advantage of it once that work has been done. If everyone always uses datagrams + demux-protocol, then that seems like a not-unreasonable outcome too.
A design decision having been present in a prior draft does not cause it to not be a source of avoidable complication. As a user of datagrams, I do not need this.
Hmm, I think the key to the statement above was the optionality of the flow ID, meaning that for use cases that did not require a flow ID, there was no additional space space taken and minimal complexity, since the behavior was exactly as it is today without flow IDs.
IIRC, support for both modes was mandatory for implementations, i.e. if you negotiated support for the extension during the handshake then you must accept flow IDs. That means it's additional implementation complexity.
NAME if both modes were brought back, then support for the flow ID extension could be made optional for implementations, so those who didn't want to implement it didn't have to. However, I think in practice, most implementations would implement it if it existed. Optional features in protocols, if sufficiently widely implemented, end up becoming de facto mandatory. And if the most common higher level protocols were to require flow IDs (e.g. H3 datagrams), then almost all implementations would implement it even if it were technically optional.
In the latest H3 QUIC DATAGRAM draft at URL (bolding is mine): provide a flow identifier allocation service. That service will allow applications co-located with HTTP/3 to request a unique flow identifier that they can subsequently use for their own purposes. The HTTP/3 implementation will then parse the flow identifier of incoming DATAGRAM frames and use it to deliver the frame to the appropriate application. I think this is an important point as to why the Flow ID should be moved to this spec instead as NAME suggests. Every single application that utilizes DATAGRAM must decide if it wants to support being "co-located" with others, and if it does it will more-than-likely be with HTTP/3 and use the Flow IDs specified there. That may be sufficient to keep from there being incompatible approaches, but the more we can do to minimize diverging implementations the better IMO. By adding it as an OPTIONAL section to this spec instead gives it increased visibility for implementers. It doesn't restrict those that definitely do not need to make use of it, but it does provide a strong reference point for those that do and is relevant to their non-HTTP/3 applications. There seems to be some consensus in this thread that such a compromise is acceptable?
I was quite surprised that this spec doesn't specify a mandatory flow ID field. It is inconsistent with the main QUIC spec where having a lot of different flows through one QUIC connection is one of the main features. I think that the flow ID must be a mandatory field in this spec, to make all the QUIC based protocols extensible by default, just to let people to mux a few different QUIC based protocols into one QUIC connection easier.
I'm still happy with having no flow ID in the frame. I've not heard a strong concrete case where the wire format needs this information. I'd like to circle back to the points on priority. QUIC is purposefully quiet on the point of signalling stream priority, this is something that every application mapping will have to reinvent, and we are happy with that design decision. Many potential clients for a feature doesn't mean we need to provide it.
I tried to put together a list of all the scenarios that I think would make use of unreliable data delivery. I came up with the following: VPN/tunneling Audio/video streaming Gaming App-layer keep alives Status updates (e.g. latest temp reading) For every one of these scenarios I can imagine some level of multiplexing being used. For that reason, I personally think QUIC should provide the (optional) multiplexing functionality natively and not require the application layers to continually reinvent the wheel. To NAME comparison to priorities, I disagree with the analogy. I don't think that all QUIC-based application protocols actually need priorities, and even few need to explicitly signal them to the peer; so keeping priority signals out of the QUIC layer makes sense. In contrast, multiplexing (whether over streams or datagrams) is a core functionality for QUIC, and I strongly believe that should extend to unreliable data delivery. So, since most application protocols would use it (IMO), we should include (as an option) a flow ID in datagrams to support multiplexing.
I agree there are application use cases that can benefit from different logical flows. But the question that is at hand is, what services does the transport provide to those flows. There has to be a reason that it exists there. Do you want flow concurrency limits, do you want per-flow buffering etc Of the adopted work I am aware of in the IETF, to make multiplexing actually work, you need an application-layer signalling mechanism. This has been achieved using HTTP/3, to suit how HTTP uses QUIC. Are the needs of other application types similar?I think a lot of the work people anticipate needs doing can't be commonly solved; we risk tying up progress on DATAGRAM by designing an abstraction for loose requirements.
I agree there are different ways to possibly treat/view multiplexed unreliable delivery. On one extreme it is a fire and forget flow identifier and (unfragmented) byte blob; At the other extremely you have "streams" with unreliable delivery, and everything that goes with it (flow ID FC; per-flow FC; fragmentation/reassembly?). I do not think the datagram extension should encompass unreliable "streams". That is a far more complicated topic and should be left to it's own extension. I do believe this extension should encompass the first scenario. Just allow for an optional flow ID. As far as on the wire efficiency, it's cheapest to do it at the QUIC layer. We just use a separate code point for datagrams with flow ID and those without (as we had originally, right?). For the application layer to implement such optional behavior would require more space. Perhaps another way to look at i:. If enough people feel strongly about wanting this, but we can't get agreement here, would you really prefer a completely different extension (and associated draft) just to duplicate what's here but adds the flow ID? IMO, that seems like overkill. Worst case, I'd prefer to have two separate TPs in this draft to indicate support/negotiation for the two different types of datagram frames.
Nick, this is a good spectrum of options / requirements. Thanks for pulling on this thread. For the record, this is my individual opinion based on my own individual implementation experience. I feel DATAGRAM is close to done and revisiting a core design decision at a late stage is unfortunate. But if its the right thing to do then it pays to really probe this and make a decision we can agree to never revisit. I don't know if everyone would agree with your pick on the spectrum. Establishing consensus on what a transport Flow ID should provide is important. Just adding a varint to the payload of the current DATAGRAM frame is easy. An application mapping can copy the blueprint of HTTP/3 DATAGRAM. I will highlight that providing more options adds some complexity to application mappings. You'll be asking them to have to decide if they want to use a flow ID or not. For example, iff a flow-ID containing transport frame were added, then H3 DATAGRAM needs to be updated to prohibit 2 of the frame types; or MASQUE need to be updated to do the same. On balance, I think the pros/cons of the current design are evenly balanced with the alternative (plus its knock on effects).
One option is to define a new frame (?) and associated TP, and just keep it logically separate from the frame and TP. HTTP/3 could continue to only use the frame/functionality, and H3/QUIC implementations could get away with not ever supporting the multiplexed variant. If we went with that approach, would you prefer a completely separate draft for that? IMO, that would entail a lot of duplication of the non-framing specific text.
I think that because there is so much overlap between the two types that, whether they are in one document or two, aapplication mappings and implementers would face the same challenge either way. So, personally, I remain to be convinced that such a design is needed. But if there is consensus to do so, a single document for both - that covers the tradeoffs - is a better way to spec it.
Makes sense. A lot of this just comes down to folks' personal opinions. As the owner of a generic QUIC stack (and not any HTTP specific code) I prefer to support it at the QUIC layer, where we already have all the multiplexing abstraction code. I also really like the "implement once" property as well. IMO it generally leads to a cleaner API design and fewer bugs overall. One final data point: I have had discussion with teams at Microsoft that have non-HTTP based protocols that are looking to move to QUIC, and I do believe they would benefit from native multiplexed unreliable delivery support in QUIC. Obviously they (and any protocol) can do it themselves, but it doesn't mean they should.
Understood. As a maintainer of a generic QUIC stack I want to know if the only service people want from me is basically variable-integer flow ID handling or if they want more. I can expose varint handling helper functions but more complicated things are harder.
As someone working on multiple applications that use DATAGRAM but not H3, I want to ensure that they all can be composed together on one connection compatibly and easily. If I implement the Flow portion of the H3 DATAGRAM independently in each application then I will also have to implement my own shim layer to manage the Flow IDs compatibly when the underlying QUIC library doesn't provide it, and hope that H3-supporting libraries also expose compatible APIs for multiplexing with HTTP. None of that is very difficult really, but it seems that a likely byproduct is pressure on DATAGRAM supporting QUIC libraries to provide the Flow ID counters in a central place independent from any H3 support. I'm not suggesting that it must be added here, but a clear (if small) specification of Flow IDs separate from the H3 DATAGRAM spec could really aid in ensuring compatible implementations and application APIs for all QUIC DATAGRAM libraries that intend to support multiple applications over one connection.
Multiplexing different application mappings inside a single QUIC connection has a bunch of complexity not confidend to DATAGRAM. For instance, QUIC provides stream IDs but the way that H3 chooses to use them means that it can't be multiplexed with other applications without some form of shim. MASQUE is one such shim, which links flow ID allocation to HTTP signalling. There's other ways to do it this of course. It sounds like NAME is asking for something similar to QUICs Stream ID management. This covers (among other things) stream ID allocation and enforcement of stream ID usage, which is a transport interoperability feature. To do the same for DATAGRAM flow IDs would require additional signalling like "maxinitialflowd" TP, and MAX_FLOWS frame. This might be something people want but I don't really know. However, even if a connection-wide transport flow ID manager existed then, just like with stream IDs, there would still be competition between application protocols that want to multiplex themselves together in the shared flow ID space. The hard problem would remain unsolved. The difficulties with sharing stream ID and flow ID space is already manifest in the WebTransport group. Demonstrating that even pooling together multiple instances of the same application mapping has challenges.
With an individual hat on - I agree with NAME on pretty much all of his points. There's definitely advantages to having a flow id at the transport, and it seems pretty straightforward at first glance. That being said there's a lot of complexity with actually managing those flow ids in a generic way. Once we put them in the transport we also have to be in the business of managing those flow ids. I agree that many applications will want to have some sort of flow id -- but I can also foresee those applications being rather opinionated about how those flow ids should behave, and I'd rather not have the transport try to meet those requirements in the generic extension.
I am better appreciating the complexity of adding even an optional basic transport-managed Flow ID, I'd rather see DATAGRAM be a simple standard as-is. If there's consensus, the simplest resolution to this issue might be to just take the paragraph mentioning HTTP/3's flow identifier and make it a ## section to draw more attention to it for application and transport implementers to consider. Possibly even expanding the text to highlight the choice applications will have to make if they want to share a connection with other DATAGRAM apps.
Discussed at IETF 110: Conversation in the room proposed that we point to H3 datagram as an example of what upper layers can do and indicate that other ALPNs should add something similar. Still didn't belong in the QUIC datagram itself, and if we find that many ALPNs repeat a pattern, they can share another extension / definition document.
I think it would be great if the text of the spec could include some brief rationale explaining the reasons for the decision reached. I think it would be helpful for readers to give some brief explanation of why the spec is the way it is on this topic, instead of having to dig through PRs / mailing list archives / etc in order to discover that.
Personally, I think an effective way of framing the rationale that NAME is seeking is a direct comparison to streams (as set out in transport section 2 and 3). Datagram is another application-data carrying primitive, highlighting the core differences could help avoid people needing to ask these questions when they puzzle about how to write application mappings. The suggestion that Eric made would be a piece of such framing.
LGTM

Revised text:
DATAGRAM frames MUST be protected with either 0-RTT or 1-RTT keys.  Note that while the max_datagram_frame_size transport parameter places a limit on the maximum size of DATAGRAM frames, that limit can be further reduced by the max_packet_size transport parameter, and by 5.1. <ins> DATAGRAM frames belong to a QUIC connection as a whole, and are not strongly associated with any stream ID at the QUIC layer.  However, it is expected that applications will want to differentiate between specific DATAGRAM frames by using identifiers, such as for logical flows of datagrams or to distinguish between different kinds of datagrams. Identifiers used to multiplex different kinds of datagrams, or flows of datagrams, are the responsibility of the application protocol running over QUIC to define.  The application defines the semantics of the Datagram Data field and how it is parsed. If the application needs to support the coexistence of multiple flows of datagrams, one recommended pattern is to use a variable-length integer at the beginning of the Datagram Data field. QUIC implementations SHOULD present an API to applications to assign relative priorities to DATAGRAM frames with respect to each other and to QUIC streams. 5.2. </ins> Although DATAGRAM frames are not retransmitted upon loss detection, they are ack-eliciting (I-D.ietf-quic-recovery).  Receivers SHOULD support delaying ACK frames (within the limits specified by a DATAGRAM frame that was thought to be lost could at a later point be received and acknowledged.  <ins> 5.3. </ins> DATAGRAM frames do not provide any explicit flow control signaling, and do not contribute to any per-flow or connection-wide data limit. frames are inherently unreliable, they MAY be dropped by the receiver if the receiver cannot process them.  <ins> 5.4. </ins> DATAGRAM frames employ the QUIC connection's congestion controller. As a result, a connection might be unable to send a DATAGRAM frame