Change:
Credit to NAME for the proposal on the .
reading this in isolation, I'm a but unsure what the "effectiveness of DPLPMTUD" is supposed to mean. Is there something implied here about how the intermediary will pack DATAGRAM frames into QUIC packets? For instance, is it being suggested that to make this work, we really need the intermediary not to pack out the size with other things?
The intent was to enable "end-to-end application DPLPMTUD" though I couldn't come up with a clear and concise name for it - in other words, there will be an application protocol that is running over HTTP Datagrams, and this is about the endpoints of that application performing DPLPMTUD at their layer. Does that make sense? If so, do you have thoughts on how best to word that?
The implication here is that the intermediary (1) knows a lower bound on the QUIC MTU and (2) never adds frames to a packet that is already larger than this bound (if it allows such packets at all). I believe this is always true for QUIC, although I'm not sure it's stated explicitly anywhere.
NAME can you clarify what you meant by "a lower bound on the QUIC MTU" ? The MTU can change dynamically due to changes outside of QUIC's control so the only real lower bound is 1200 - and we need to be able to send packets larger than that.
I mean that, at any given time, a QUIC endpoint has a lower bound L on the estimated path MTU. This lower bound is sometimes wrong, but it's never zero. If an H3 DATAGRAM produces a QUIC packet that is larger than L, we just need to be sure that QUIC won't add other frames to this packet. (I don't know if QUIC will send this packet or just drop the datagram, but either way, DPLPMTUD on the inner connection will work fine.)
The in the PR sounds fine. I'm still confused by the justification though, even after these comments. For hop2, which is QUIC, it must have a minimum PMTU of 1200. The maximum PMTU could be bigger. The maximum QUIC packet size that can be sent should be the min(max MTU, maxudppayload_size advertised by peer). After the intermediary gets the H3 DATAGRAM payload, it will need to reencode it. We're suggesting into a QUIC DATAGRAM frame, with H3 DATAGRAM headers. This reencoding could have different overhead sizes (hop2 might have larger QUIC packet overheads, or H3 DATAGRAM overheads. Is the suggestion that the intermediary is using a PMTU size P1, and if it sees a H3 DATAGRAM that would produce a packet larger that P1, it should try to send it as a sort of probe?
That's fine, so long as the overhead of each hop rarely changes. It could try to send it, or it could drop it. (It's not much of a "probe", since the intermediary won't necessarily know if it was dropped.) Either way, the end-to-end DPLPMTUD will converge to some working value, which is all that matters.
So, can we remove this statement wrt to DPLPMTUD? I don't know what I'm supposed to apply it to.
I'm OK removing it, but in general I think it would be helpful to have the draft explain why that SHOULD NOT is there. And the motivation is to allow e2e DPLPMTUD. Because if the intermediary reencodes as capsules some of the time, the application-endpoints won't be able to get a sense for what packet sizes get through or not.
What you just said is much clearer than the proposed text. Longer term, I think what we'll probably end up with is a whole section that collects intermediary concerns. But until we arrive at that point, a better way to present the matter at hand could be something like the following, which is intended to follow on the theme of discussion from the paragraphs in that section. "Intermediaries that receive HTTP Datagrams in QUIC DATAGRAM frames need to decide how to forward them. If the onward connection supports QUIC DATAGRAM frames, intermediaries SHOULD attempt to forward HTTP Datagrams in a QUIC frame. If the HTTP Datagram is too large to fit in a QUIC frame, the intermediary SHOULD drop it. This preserves the end-to-end unreliability characteristic that methods such as DPLPMTUD depend on. An intermediary that forwards HTTP Datagrams in DATAGRAM capsules allows HTTP Datagrams to be arbitrarily large. This can misrepresent the true path properties, defeating methods such a DPLPMTUD."
I like that, NAME I've updated the text based on your comment, let me know what you think.
So this is the first step to drop them. However, I am fairly convinced that we also need the signal to the Intermediaries when they need to send a HTTP datagram over the reliable stream instead of over the QUIC Datagram. Otherwise we will have MTU black holes for the minimal IPv6 packet size.
To expand on my previous comment. At least in CONNECT-IP and I think also for CONNECT-UDP. We do need a solution to ensure that we can tunnel and not blackhole inner payloads that correspond to the IPv6 minimal packet size. So for Connect-IP with all the overhead we need to send HTTP datagram containing a payload for 1280 IPv6 packets, otherwise we are not IPv6 compliant. For CONNECT-UDP this is not as clearly required as in the case of IPv6, however I think it would be a tradgic if this solution couldn't handle the required smallest allowed size of QUIC initial packets. That is why I think we need to have a solution where the endpoint can be certain that even if less efficient that the packets make it through if they are below these minimal sizes.
This sounds like a form of fragmentation to me. I'd rather fail hard than be fooled by unknown intermediaries that the path supports the size required. Instead of signalling the explicit MTUs as suggested on the mialing list, what about a capsule type equivalent to PTB that could be returned to the client to indicate something was too large and dropped. Then the client can make the active decision whether to reduce size (if possible) or retry by putting the HTTP Datagram in a DATAGRAM capsule itself.
NAME that I think works for me. In fact I think that is a better solution then the boundary value which requires state keeping and allows the HTTP endpoint to decided what to do. Also it can provide an endpoint with the actual current MTU. It is only executed when you can't forward the HTTP Datagram so it appear simpler than my original idea which would have needed cross layer interaction on changes in lower layers independent of when it occurs. But, yes this is fragmentation and reassembly. Just the matter that we have the facilities available in the form of QUIC streams or TCP.
I think a PTB reply is likely unnecessary complexity. Anything on CONNECT-IP has to exit the tunnel eventually, and it still needs PMTUD on that path. ICMP connectivity is not reliable, so PLPMTUD is mandatory regardless, and will automatically handle any bottlenecks in the tunnel without the need for PTB.
just to clarify a little, I didn't mean a literal PTB. I meant to address Magnus' point that an end-to-end QUIC connection could fail to even get an client Initial through because the intermediary willingfully dropped it. A client might benefit from an immediate signal from the intermediary in such a case. I'm not sworn to this idea though.
Note that in CONNECT-UDP, the Initial should almost always travel in a CAPSULE, so it can be sent immediately ("false start") without worrying about reordering or packet loss. That has ... interesting implications for MTU.
interesting, I hadn't thought of doing that. Right now my implementation buffers received datagrams, so whether I receive the DATAGRAM frame before a corresponding HEADERS is negligible. Agree it might have interesting implications.
NAME I don't think it is sufficient for us to simply declare that it is someone else problem with the MTU. From an Internet Architecture perspective the MASQUE tunnel is the equivalent of a link layer. There are certain requirements on link layers that carry IP packets, especially IPv6 packets. This requires support of the 1280 MTU, so we need a solution for that case. Secondly, I am a bit intrigued by your statement that in the case of Connect-UDP the first UDP packets would travel as capsule. I don't think the HTTP Datagram mandates that in any way. To my understanding, which appears to match NAME ones. Is that one can send a Connect-UDP request, followed by a HTTP Datagram context registration capsules for the needed Context IDs and then send HTTP Datagrams. There is nothing in the protocol that prevents this. Our Connect-UDP implementation will buffer these early HTTP datagrams and provide them to the Masque Server when it has processed the registration to create the IP/UDP packets based on the payload.
Hi folks, as per let's have this discussion somewhere that's not a GtiHub PR. I'd suggest . Since I haven't heard any objections on the text in this PR, I'm going to merge it. Let's discuss on the list whether we think more should be done about MTU or not.
This has been sent to the MASQUE WG mailing list in hopes to get some discussion there. However, I want an issue that tracks this: In the work of writing up our Connect-IP proposal (URL) we looked into how to deal with MTU issues effectively. Our conclusion was that this is going to be a general problem for any user of HTTP Datagrams. Thus, we like to propose that MTU handling is done within HTTP datagram. This email will start explaining what we see as requirements for a MTU signaling solution for HTTP datagram, then propose a potential solution. Lets start with a figure that provides us with a framework to discuss the requirements: I think what makes this a bit more complex is the fact that we need to consider HTTP intermediaries, such as a front end load balancer that terminates a first QUIC and HTTP/3 connection between the client and that intermediary. From that intermediary another HTTP connection is used towards the HTTP server that consumes and produces the HTTP datagrams, and in the case of CONNECT-UDP and CONNECT-IP it also have a third path towards the target to consider. This figure includes two clients to remind us to consider that the HTTP intermediary may actually aggregate the HTTP request and response and HTTP datagram over one HTTP/2 or HTTP/3 connection over Path . In the case of MTU this complicates things more as the proxy cant assume that all HTTP requests have the same MTU for its datagrams on a HTTP/3 connection. And the Intermediary will be the entity that have direct knowledge of the client facing as well as the next hop MTU over the HTTP connections that may all differ in MTU. We also have to consider the fact that the underlying transport connection may at any time be subject to a IP MTU change due to route change for the path between the nodes. In addition if one have enabled PMTUD in TCP or QUIC a larger MTU on the individual path could be made available and in some case desirable to use. Thus, we need to consider dynamic changes during the HTTP connections life time and each HTTP request response pairs usage of HTTP datagram. So when using HTTP/3 datagram there are a strict MTU limit on the individual datagrams for it to be possible to be sent as QUIC datagrams, and not being forced to be encapsulated as CAPSULES over the reliable stream. This is clearly a possibility but results in that the datagrams are sent reliable and in order for each HTTP request, i.e. Connect-UDP or CONNECT-IP request. Also in case some end-to-end payloads fit in HTTP datagrams and others don’t there is potential for reordering among the payloads. Thus, to avoid this the client and the proxy needs to determine what the lowest currently supported HTTP datagram size on the path. For each QUIC connection the end point will know what the initial MTU value is for this path when the HTTP/3 connection has been successful completed. However, that knowledge will not be available if one attempt to construct and send the HTTP request prior to connection establishment has concluded. So the requirements we see for an MTU handling solution for HTTP Datagrams are the following. 1) Hop-by-hop signaling across the HTTP entities of the lowest MTU of any sub-path 2) Needs to be associated to a particular HTTP request or end-to-end path to support aggregation by HTTP intermediaries 3) Endpoints needs to be able to initiate update of the MTU value upon detection of any changes during HTTP Datagram streams lifetime. 4) HTTP Intermediaries needs to be able to initiate updates of the MTU value upon detecting MTU changes from the individual HTTP connections. Solution proposal A new HTTP Datagram Capsule is defined for MTU value exchange. This one is intended for the HTTP intermediary that needs to interpret, update or initiate sending of it. Thus, it needs to be a fixed registered type so it can be easily processed. It can also be exchanged in parallel to the RegisterDatagram* capsules as at that point the underlying HTTP connection is established and initial HTTP Datagram values will be known. A capsule will also travel all the way to the end. And an intermediary can initiate one in each direction for request paths. The only downside I see of this is that one is required per open stream when MTU changes occurs. Maybe someone have an idea of how to handle signaling when aggregating multiple endpoints streams onto one HTTP/3 connection. To make it more efficient, rather than sending one MTU capsule per stream, an MTU capsule could list all streams it is applicable to. That way the number of MTU capsules would be no more than the number of end-to-end paths actually used.
Hi NAME thanks for proposing this work item. From reading your problem statement and proposal, this will require non-trivial amounts of complexity. Resolving that will take time. I don't see a reason to delay the HTTP Datagrams draft because of this, so I'd suggest building this as an extension to HTTP Datagrams. I think this would even be a great way to test out our new extensibility mechanism! Can you write this up as a separate draft please?
Yes, it is non-trivial. However, I think we need to settle if we have a day one need for this mechanism to ensure that CONNECT-UDP actually can work properly and handle QUIC in QUIC when the outer tunnel uses minimal QUIC packets. Similarly we have a need for CONNECT-IP to handle IPv6 minimal packet size. So I think we need to discuss if this needs to be solved. But, yes I am willing to write this up in spec form, but I was more thinking of a PR to the HTTP-DATAGRAM draft.
I don't see a day one need here that isn't addressed by . Given the current text, CONNECT-UDP (and -IP) endpoints can detect datagram blackholing and switch to capsules.
I don't understand the following statement "CONNECT-UDP (and -IP) endpoints can detect datagram blackholing and switch to capsules." I don't believe that is true in scenarios with HTTP Intermediaries as the MASQUE endpoint in those case lacks visibility of the dropping of the HTTP Datagram, unless additional signals are added. But, I would also prefer that we discuss this on the mailing list. I just think closing the issue was premature.
You don't need an explicit signal to know that a packet was dropped, you can infer that through lack of acknowledgements. Since there aren't acknowledgements at the datagram layer itself, you'd need to partner with the transport protocol running over CONNECT-UDP or CONNECT-IP to get that signal. That isn't trivial, but if we transparently solve this at the datagram layer (let's say we add a "datagram too big" capsule and then have the datagram layer use it to switch to reliable capsules for example) then you'd end up with terrible performance because in most scenarios you'd be much better off with the higher level transport protocol using a smaller MSS that fits in datagrams. So all this to say: this isn't an easy problem to solve, and having an explicit signal isn't a silver bullet that solves it easily. Since we need to involve the higher layers, then we have many options to pick from, and not all of them require an explicit signal.
Agree that for sufficiently large MTU you want the end-to-end PMTUD to work and dropping will be necessary. However, we are going to have requirements on minimal packet sizes, that is where we end up in needing a signal. In addition if the tunnel HTTP Datagram ingress knows the MTU, then for the MASQUE use cases it can send back the appropriate signal towards an remote IP point or through the API to the application using MASQUE. If an HTTP intermediary simply drop packets that are to large and send no signal back. Then the HTTP datagram ingress has no knowledge and no type of signal intended to speed up PMTUD is also sent. Which, makes results us from potentially breaking the IPv6 specification requirements in two ways when using it for CONNECT-IP. First we may not support the required datagram size of 1280 bytes. Secondly, we are not supporting the IPv6 PMTUD model of actually sending ICMPv6 PTBs. Yes, this may not work as well as it should, but it is still required functionality.
This sounds like a good use case for . It's a great example of a context ID extensions that can benefit multiple methods. As long as draft-ietf-masque-h3-datagram is extensible enough to allow this kind of extension, I don't think we need to solve this in draft-ietf-masque-h3-datagram.
Summarizing the we had on this topic, both WebTransport and CONNECT-UDP are working well in production without an MTU solution at the HTTP Datagrams layer. There is also no proposal for an MTU solution that works for all possible use cases of HTTP Datagrams. The only unsolved problem here is how to reconcile a low HTTP Datagram MTU with the IPv6 minimum link MTU. Since that's specific to CONNECT-IP, I've filed a specific issue for that there: URL Based on this, I'm going to close this issue with no action. If you disagree, please reopen/comment. Hi, In the work of writing up our Connect-IP proposal (URL) we looked into how to deal with MTU issues effectively. Our conclusion was that this is going to be a general problem for any user of HTTP Datagrams. Thus, we like to propose that MTU handling is done within HTTP datagram. This email will start explaining what we see as requirements for a MTU signaling solution for HTTP datagram, then propose a potential solution. Lets start with a figure that provides us with a framework to discuss the requirements: +--------+ PathA +--------+ +-------+ +--------+ +--------+ +--------+ PathB +--------+ +--------+ +-------+ +--------+ I think what makes this a bit more complex is the fact that we need to consider HTTP intermediaries, such as a front end load balancer that terminates a first QUIC and HTTP/3 connection between the client and that intermediary. From that intermediary another HTTP connection is used towards the HTTP server that consumes and produces the HTTP datagrams, and in the case of CONNECT-UDP and CONNECT-IP it also have a third path towards the target to consider. This figure includes two clients to remind us to consider that the HTTP intermediary may actually aggregate the HTTP request and response and HTTP datagram over one HTTP/2 or HTTP/3 connection over Path . In the case of MTU this complicates things more as the proxy cant assume that all HTTP requests have the same MTU for its datagrams on a HTTP/3 connection. And the Intermediary will be the entity that have direct knowledge of the client facing as well as the next hop MTU over the HTTP connections that may all differ in MTU. We also have to consider the fact that the underlying transport connection may at any time be subject to a IP MTU change due to route change for the path between the nodes. In addition if one have enabled PMTUD in TCP or QUIC a larger MTU on the individual path could be made available and in some case desirable to use. Thus, we need to consider dynamic changes during the HTTP connections life time and each HTTP request response pairs usage of HTTP datagram. So when using HTTP/3 datagram there are a strict MTU limit on the individual datagrams for it to be possible to be sent as QUIC datagrams, and not being forced to be encapsulated as CAPSULES over the reliable stream. This is clearly a possibility but results in that the datagrams are sent reliable and in order for each HTTP request, i.e. Connect-UDP or CONNECT-IP request. Also in case some end-to-end payloads fit in HTTP datagrams and others don't there is potential for reordering among the payloads. Thus, to avoid this the client and the proxy needs to determine what the lowest currently supported HTTP datagram size on the path. For each QUIC connection the end point will know what the initial MTU value is for this path when the HTTP/3 connection has been successful completed. However, that knowledge will not be available if one attempt to construct and send the HTTP request prior to connection establishment has concluded. So the requirements we see for an MTU handling solution for HTTP Datagrams are the following. Hop-by-hop signaling across the HTTP entities of the lowest MTU of any sub-path Needs to be associated to a particular HTTP request or end-to-end path to support aggregation by HTTP intermediaries Endpoints needs to be able to initiate update of the MTU value upon detection of any changes during HTTP Datagram streams lifetime. HTTP Intermediaries needs to be able to initiate updates of the MTU value upon detecting MTU changes from the individual HTTP connections. Solution proposal A new HTTP Datagram Capsule is defined for MTU value exchange. This one is intended for the HTTP intermediary that needs to interpret, update or initiate sending of it. Thus, it needs to be a fixed registered type so it can be easily processed. It can also be exchanged in parallel to the RegisterDatagram* capsules as at that point the underlying HTTP connection is established and initial HTTP Datagram values will be known. A capsule will also travel all the way to the end. And an intermediary can initiate one in each direction for request paths. The only downside I see of this is that one is required per open stream when MTU changes occurs. Maybe someone have an idea of how to handle signaling when aggregating multiple endpoints streams onto one HTTP/3 connection. To make it more efficient, rather than sending one MTU capsule per stream, an MTU capsule could list all streams it is applicable to. That way the number of MTU capsules would be no more than the number of end-to-end paths actually used. Cheers Magnus Westerlund
I agree that MTU discovery is valuable, and I think we should try to provide a workable solution. However, I think the solution you've sketched sounds very complicated, and I have some concerns about whether it would be implemented correctly. Here's an alternative proposal: An HTTP intermediary routes each datagram from an "incoming" link to an "outgoing" link. When the outgoing link is DATAGRAM-capable, intermediaries SHOULD NOT convert DATAGRAMs to CAPSULEs, and SHOULD NOT fragment (or permit fragmentation of) the corresponding QUIC packets. DATAGRAMs that are too large for any segment's MTU will therefore be dropped silently, and will not reach their destination. This is sufficient to enable end-to-end PMTUD over MASQUE. I think this arrangement is likely preferable: it avoids adding complexity to intermediaries, and places responsibility for PMTUD at the endpoints, who have the clearest incentive to make it work well. Endpoints who need to send packets larger than the MTU, or who need to send large packets before PMTUD converges, can use CAPSULEs at their discretion. >

Discussion:
applications have to always consider HTTP Datagrams to be unreliable, even if they were initially sent in a capsule. <ins> If an intermediary receives an HTTP Datagram in a QUIC DATAGRAM frame and is forwarding it on a connection that supports QUIC DATAGRAM frames, the intermediary SHOULD NOT convert that HTTP Datagram to a DATAGRAM capsule.  If the HTTP Datagram is too large to fit in a DATAGRAM frame (for example because the path MTU of that QUIC connection is too low or if the maximum UDP payload size advertised on that connection is too low), the intermediary SHOULD drop the HTTP Datagram instead of converting it to a DATAGRAM capsule.  This preserves the end-to-end unreliability characteristic that methods such as Datagram Packetization Layer Path MTU Discovery (DPLPMTUD) depend on RFC8899.  An intermediary that converts QUIC DATAGRAM frames to DATAGRAM capsules allows HTTP Datagrams to be arbitrarily large without suffering any loss; this can misrepresent the true path properties, defeating methods such a DPLPMTUD. </ins> 5. In order to facilitate extensibility of contexts, the