jiebi/CodeConvo huggingface dataset:
This PR aims to take the design discussions we had at the last interim, and propose a potential solution. This PR should in theory contain enough information to allow folks to implement this new design. That will allow us to perform interop testing and confirm whether the design is sound.
I'm confused by this pull request (probably because I was out of the loop for a while). My understanding, at least when I first read the proposal a month ago was that we would separate multiplexing streams and multiplexing contexts into different layers, so that methods that need context would get it (CONNECT-UDP), and features that don't (WebTransport/extended CONNECT) could use stream-associated datagrams directly.
NAME you're referring to which this PR isn't trying to address. That'll happen in a subsequent PR.
In , capsules, contexts and context IDs are explicitly defined as end-to-end and there are requirements for intermediaries to not modify them, and to not even parse them. The motivation for this was to ensure that we retain the ability to deploy extensions end-to-end without having to modify intermediaries. NAME mentioned that these requirements might be too strict, as he would like to have his intermediary be able to parse the context ID, for logging purposes.
Part of the comment from the PR is also about consistency of interface for generic HTTP libraries that can be used to build either endpoints or intermediaries. If the context ID is always present, it's overly prescriptive to say that an intermediary can't parse it, and such a requirement couldn't be enforced anyways. If context IDs are negotiated on/off, whether this is possible depends on the negotiation mechanism.
I think its fine for a component that has bytes passing through it to look at those bytes. The import thing to do is set expectations on what that means for the protocol behaviour we are defining. We don't expect intermediaries to do anything with a context ID; they should not enforce the protocol semantic rules (odd, even, reuse etc). H2 and H3 somewhat solve this problem by decoupling frame parsing from frame payload interpretation. E.g. if the form of a request or response is invalid, an intermediary shouldn't pass it on. DATAGRAM is different because it isn't defined as an actual HTTP/3 frame with frame headers and frame payload.
Something that might work is to formalise that quarter stream ID is part of the envelope that intermediaries have to be able to handle the syntax of, but context ID is part of content and separate.
Unless you encrypt capsules end-to-end, I don't think any text in the spec can in practice prevent intermediaries from inspecting and modifying capsules. You can send something like GREASE capsules, though.
In , we introduce the CLOSEDATAGRAMCONTEXT capsule which allows endpoints to close a context. That message currently only contains the context ID to close. NAME suggests that we may want to add more information there. Should we differentiate between "context close" and "context registration rejected"? Should we add an error code? In particular, do folks have use cases for these features?
My first concern is with infinite loops. If the context was "garbage collected" by the recipient, the sender can simply re-create it if they still want it. However, if it was closed due to incompatibility (i.e. rejected) or hitting a resource limit (e.g. max # of contexts), reopening it in response will produce a tight infinite loop. This could be maybe be avoided by some kind of heuristic, but an explicit indication of what's going on seems better.
This issue assumes that decide to go with the two-layer design described in . Given that design, some applications might not need the multiplexing provided by context IDs. We have multiple options here: Make context ID mandatory Applications that don't need it waste a byte per datagram Negotiate the presence of context IDs using an HTTP header Context IDs would still be mandatory to implement on servers because the client might send that header Have the method determine whether context IDs are present or not This would prevent extensibility on those methods Create a REGISTERNOCONTEXT message This assumes we use the Message design from We add a register message that means that this stream does not use context IDs This allows avoiding the byte overhead without sacrificing extensibility
Making the context ID mandatory is simple, although slightly wasteful. For the variations where it is optional, I prefer the one that uses a header. It could work with a variant of where we do Header + Message, and the Header determines whether or not you have contexts, and whether or not they can be dynamic.
What are the benefits of message over header to make it optional?
To me it seems like header is sufficient, because the context IDs are end-to-end, and should not affect intermediaries, making the header a natural place to put it. I think having the context ID be the default behavior and the header is to opt-out is the right decision if we were to allow removing it.
This is a bit of a bike-shed, but why would it default to having context-IDs instead of not having them?
I think the second option "Negotiate the presence of context IDs using an HTTP header" is the best personally. In terms of: "Context IDs would still be mandatory to implement on servers because the client might send that header", I assume server in this context is the final application server, not a proxy? Even so, I'd argue it's applications which decide if they need this functionality, so if a server only implemented one application and it didn't use Context-IDs, then I don't see why it would need to implement it.
Ian, in your comment, are you talking about an intermediary when you say "proxy"? If so, then an intermediary need not implement anything based on the payload of than H3 DATAGRAM, it only has to pass the data along. That may be worth a clarification.
Having context ID optional has a nice future-proof property - other uses of datagrams in HTTP/3 can decide if they want the second layer or not. I wouldn't even state it as application needing multiplexing or not. Other applications might find it very useful to include a second ((third, fourth, fifth) layer of frame header. I think its fine for us to state as much in this in the spec, while also defining an explicit second layer for multiplexing.
Yes, by proxy I meant intermediary. And I agree about what intermediaries need to do, I just wanted to ensure I understood what server meant when David used it above. Yup, whenever all this gets written up, we can make it clear that intermediaries can ignore much of this.
The method defines the semantics of datagrams for a particular request, thus I think the third option makes most sense.
I don't see a point in using explicit negotiation, since in the two cases I can think of the top of the head, the answer is always clear from the method (CONNECT-IP seems to always want context IDs, WebTransport does not ever want context IDs). If some method starts out without context, but wants to add context later, it can add its own negotiation.
One set of use cases for which the method approach(3) is best are those where there are multiple context IDs but they're predefined by the method and don't need negotiation. My strawman is WebRTC.
During the 2021-04 MASQUE Interim, we discussed multiple options for registering context IDs. Without going into the specific of the encoding, there are two broad classes of solutions: "Header" design Once-at-setup registration This happens during the HTTP request/response Simplest solution is to use an HTTP header such as (encoding TBD) Example use-case: CONNECT-UDP without extensions "Message" design Mid-stream registration This can happen at any point during the lifetime of the request stream This would involve some sort of "register" message (encoding TBD, see separate issue) Example use-case: CONNECT-IP on-the-fly compression (this can't be done at tunnel setup time because the bits top be compressed aren't always known at that time) It's possible to implement once-at-setup registration using the Message design, but it isn't possible to implement mid-stream registration. Therefore I think we should go with the Message design. It would also be possible to implement both but I don't think that provides much value.
I'd like to here some more from the WebTransport use case on the issue. My understanding is that the Requests are used to create WebTransport sessions and that creation of datagram flows(contexts) could be a lot more ad-hoc / server generated after the request phase, when compared to the CONNECT-UDP case. Cc NAME
Personally, I'd vote for either "message" or "message + header" (a la priorities), but not just "header". Mid-stream annotation of context/flow/etc is quite useful.
I prefer "Message" only, since the dynamic use case seems like a fairly compelling use of sub-flows(contexts?).
I'm not sure there are many cases when one allocates all sub-flows up front AND needs to use a header to do that in a flexible way. If an application needs multiple sub-flows, but they're fixed in number/format(ie: data and control), I don't think there's a need for a header or a "Message", since the application can consider that part of its internal format.
During the 2021-04 MASQUE Interim, we discussed having the ability to accept or reject registration of context IDs. An important motivation for this is the fact that context ID registrations take up memory (for example it could be a compression context which contains the data that is elided) and therefore we need to prevent endpoints from arbitrarily making their peer allocate memory. While we could flow control context ID registration, a much simpler solution would be to have a way for endpoints to close/shutdown a registration.
IIUC there's an opposing view that the constraints can be negotiated apiori. Either statically (such as defining an extension which states that there can only be N contenxts of a given type) or dynamically using something like a HTTP header. Failure to meet these constraints could lead to an HTTP/3 stream or connection error rather than a flow rejection.
NAME I've filed to discuss whether to register/negotiate at request start or mid-stream.
I think that if there is the ability to register mid-stream (), we need the ability to close/de-register/reject mid-stream as well. If we don't have mid-stream changes, then no need to negotiate. However, someone trying to register something needs to be able to know whether or not the thing they registered will be accepted.
I agree that notification of the registration status is particularly important if we allow mid-stream registration.
In draft-ietf-masque-h3-datagram-00, flow IDs are bidirectional. During the 2021-04 MASQUE Interim, we discussed the possibility of making them unidirectional. Here are the differences: Bidirectional single shared namespace for both endpoints (even for client, odd for server) it's easier to refer to peer's IDs, which makes negotiation easier Unidirectional one namespace per direction every bidirectional use needs to associate both directions somehow I'm personally leaning towards keeping them bidirectional.
Unlike QUIC streams, these do not have a state machine or flow control credits, so there is no cost to only using a single direction of a bidirectional flow for unidirectional use-cases.
There's another option where flows are unidirectional but they share a namespace (cleint even/server odd). I'm undecided. And I also don't know how important it is to choose. The 2 layer design means that DATAGRAMs always have a bidirectional tie via the stream ID. On balance, I think bidirectional would be good enough today but I would be receptive to extension use cases that make a strong point of unidirectional.
Bidirectional is my personal preference, but either way could work ultimately. It seems to me that any extension that wants unidirectional can just only use this context/flow/etc ID for one direction if it wants.
I think bidirectional flows are likely overcomplicated and unnecessary. For example, if either peer can "close" a flow, this requires a much more complex state machine. In my experience, half-closed states are hard to get right. Closing unidirectional flows seems much easier to reason about. Many, perhaps most, of our use cases for flows are really unidirectional. For example, DSCP marking likely only makes sense from client to server, while timestamps for congestion control tuning likely only make sense from server to client. QUIC connection IDs are different in each direction, so QUIC header compression may be easiest to understand with unidirectional flows.
Bidirectional doesn't imply half-close states. I'm thinking of a bidirectional flow that can only be closed atomically as a whole. I don't think that's true. The default extension-less CONNECT-UDP is bidirectional. So is ECN, and so would a timestamp extension.
I would prefer bidirectional without support for half-close. I feel that having unidirectional flow/context IDs adds overhead we don't need; peers now need to track those associations and it's unclear what the benefits of having them separated out are.
This issue is conflating consent (unilateral/bilateral) with scope (unidirectional/bidirectional). IIUC there are two questions here: 1) Can I send a flow-id forever once declared, or can the endpoint tell me to stop without killing the whole stream (consent)?
2) Can the receiver of a flow-id assignment use that same flow-id when sending datagrams, or could this mean something completely different? I think the answer to (1) is "yes, it can tell me to stop" (i.e. it is negotiated while allowing speculative sending). For (2), I lean toward bidirectional, but not strongly. There are plenty of codepoints, so it seems wasteful for each endpoint to define the same semantic explicitly.
NAME the intent of this issue wan't to conflate those two things. This issue is about scope. For consent, see issue .
Conceptually, I like unidirectional better. I tell the peer that when I send with context-ID=X, it has a given semantic. For bidirectional the sender is saying both that and that if the peer sends on that ID the original sender will interpret with the same semantic. That might not even make sense. Is it possible to associate datagrams with a unidirectional stream (push?). If so, does allowing for bidirectional context imply that even though the receiver cannot send frames on the stream, it can send datagrams? Every negotiation does run the risk of loss/reordering resulting in a drop or buffering event at the receiver. The appeal of bidirectional is that it removes half of this risk in the cases where the bidirectionality is used.
To second NAME I would also prefer bidirectional without support for half-close. Given you can always not send something in one direction, and there's none of the overhead of streams(ie: flow control, etc) I can't see any benefits of unidirectional over bidirectional in this case. Associating h3-datagrams with a server push seems like unnecessary complexity to me, but you're right that it should be clarified.
During the 2021-04 MASQUE Interim, we discussed a design proposal from NAME to replace the one-layer flow ID design with a two-layer design. This design replaces the flow ID with two numbers: a stream ID and a context ID.
The contents of the QUIC DATAGRAM frame would now look like: While the flow ID in the draft was per-hop, the proposal has better separation: the stream ID is per-hop, it maps to an HTTP request the context ID is end-to-end, it maps to context information inside that request Intermediaries now only look at stream IDs and can be blissfully ignorant of context IDs. In the room at the 2021-04 MASQUE Interim, multiple participants spoke in support of this design and no one raised objections. This issue exists to ensure that we have consensus on this change - please speak up if you disagree.
I am fine with the 2-layer design for CONNECT-UDP, but I am not sure if the second identifier should be in H3 DATAGRAM or should be specific to CONNECT-UDP. As I indicated to the list, webtrans and httpbis might weigh in on this with broader consideration of other use cases.
Hi NAME the topic of the optionality of context IDs is discussed in issue .
I think this is a good design, because it minimizes the core functionality of the draft, but avoids every application that needs multiple dynamically created sub-flows from re-inventing a different mechanism. Nit: I would not call this a 'Frame', since it's not an HTTP(TLV) frame.
I think this is a good change. Thanks! There are some things that I think we should still discuss, but we shouldn't hold up this change on those details.
Several small updates, mostly to clean things up a bit, and align property names and such.
NAME - I think you're wrong: my understanding (and also the way the text is written - otherwise we'd have to change it in the API draft) was that Connection Properties can (and should) also be set on Preconnections, to give as much input as possible for Connection establishment. In the API draft, the Selection Properties in Section 5.2 can only be used on Preconnections: you can't ask for congestion control after establishment. You can't say that you'd rather have multistreaming support after establishment. But that doesn't exclude Connection properties from being usable earlier. Required minimum checksum coverage could influence choosing UDP-Lite vs. UDP. Bounds on Send or Receive Rate could influence the interface choice. The capacity profile could influence various things (and no, there isn't a matching Selection Property now - there is a per-Message override). So... I think that either you're wrong, or I'm missing something and the API draft heavily lags behind a decision that was made. I hope not!
NAME The API drafts says: I interpret this as only the Selection Properties influence selection and the Connection properties are used to further configure the connections according to user preferences. Otherwise what is the difference between them?
If you interpret it as you do, I think the text isn't clear enough (indeed it isn't super clear - in particular, "eventually established" is misleading IMO). Though, note that it says that Connection properties can also be used later. Regarding the difference between them: Selection Properties only make sense for Preconnections.
NAME - sorry lost this thread. But I saw that you have started cleaning up some Connection properties that had mixed Connection and Selection properties up in the API draft. Does it mean that we are now in synch and agree that only Selection properties are used in the selection and Connection properties are used for configuration (both during establisment and also after the connection setup).
NAME the other PR means nothing about what I said here: it only tried to address issue which is about a wording inconsistency in a specific property. You say that Connection properties can also be used during establishment - if you agree with allowing this, then why not allow to configure them as early as possible and allow them to influence selection?
What's bad about having more information available that can guide the selection process?
NAME the problem is that the Connection properties are not expressed in a way suitable for selection. If they should also be used for Selection they would have to be reworked to be of type Preference. Otherwise I do not know what they mean for selection. What does it mean if I pick say a particular connection group transmission scheduler? Is it required or preferred? Most connection properties also do not look very useful to me for selection so in case there are some we need I think it would be simpler to just define a matching selection property.
NAME we're only discussing whether it's allowed to use them as additional input or not. I agree, some (like the scheduler choice) don't seem useful, but some do... generally we may not have good guidance on how they should be used by an implementation, but giving the system freedom to use as many inputs as possible is probably a good thing. Regarding the type, require/prefer//ignore/avoid/prohibit obviously works fine for a "wishlist" of boolean character: having something, or not having it. The selection properties are generally like that - except "Interface Instance or Type" and "Provisioning Domain Instance or Type" which are, well... wishlist-collections. Consider the capacity profile. Surely configuring this early can be useful input - e.g. if someone picks low latency / interactive, you may even want to put this traffic on a specific network interface! But if you'd want to make a copy of the capacity profile as a "proper" selection property, how would you do it? Apply require/prefer/ignore/avoid/prohibit for each of its values? Then I could say "prefer" low latency/non-interactive but "avoid" low latency/non-interactive, which would surely be odd. Apply the qualifier to the whole thing? Then it's about, e.g. "avoiding" to use a capacity profile at all. That's odd as well. I think it's in the nature of the current selection properties that they merely say "I want to be able to do this, and this is how important it is to me", which is different from configuring something. That's also why, as written, they simply wouldn't work for later connection configuration.
Connection properties really are about configuring something, but I still say that being able to use them as early input is good. I agree that the implementation isn't super clear as they don't come with a require/prefer/... etc. qualifier, but then we just offer some freedom to implementers.
NAME I just pushed a commit that updates the selection and connection properties text, removing "connection" almost everywhere, but adding a disclaimer in the beginning that says that connection properties should be configured as early as possible because they can influence decisions made by the implementation, and "In the remainder of this document, we only refer to Selection Properties because they are the more typical case." Frankly I'm not sure this is better... but it does remove most of the "selection and connection" double mentions that you pointed out.
Fixed up branch building and resolved merge conflicts. I'll merge this in later today before the posting deadline.
Cool thanks! Sorry if I messed something up
As discussed in
Thank you for updating this – looks good! just needs s/send-params/msg-properties/ and merging