Draft: draft-ietf-quic-manageability

Commentary: This is my suggestion to 
 Good question . I think it 'd be unwise to take this action ( drop & send ICMP ) blindly without regard to how the transport protocol will react . UDP of course carries a diverse set of services , and not all of them will react well to this . If all implementations complied with RFCs there should be no issue ; a DF bit set means the sender is expecting it might have to deal with drops . Hopefully they have tested that it works gracefully . But from a sense of caution I prefer to take the approach that such actions should be targetted on services where we have more confidence that it will react well . The finalised QUIC is one such transport , and significantly it 's one that will carry the majority of internet traffic before long . 
 I find reasoning pretty convincing . The on path equipment is already taking different approaches based on the transport protocol ( i.e. MSS clamping with TCP ) and allowing some existing protocols to be fragmented . QUIC being treated as the latter category is obviously problematic , so the operator has two choices that would allow DPLPMTUD to function for QUIC : doing any sort of fragmentation . a best effort attempt to not fragment QUIC packets and rather drop them on the floor . The former is likely to disrupt some protocols which are functioning " fine " today with fragmentation and reassembly , which may make it untenable for an operator . The latter , while not totally straightforward and smells of ossification , seems like it would be more tenable in practice while addressing the actual issue . 
 What is still not clear to me is why is the advise here " please drop big QUIC packet ( rather than fragment ) " instead of directly saying " please drop if DF is set " ( which I think is the advise we already have in various RFCs ) . If operators do n't follow the later advise now , why should they follow the former ? 
  , it is true that " please drop if DF is set , and send PTB " is existing advice that is not universally obeyed . And the advice in this PR will not be universally obeyed either , but some operators will take note , and some developers of PGW software will write code to implement it . ( This is not a natural function for a PGW , because the MTU on the S1 - U side is large . It has to be deliberately configured based on knowledge of the downstream network , just like MSS clamping currently is . ) You 're right to ask why should the implementations be focussed on QUIC , rather than what it should be from an architectural purity perspective , which is all IP that has a DF property . Here my answer is that any network operator contemplating deliberately preventing some traffic that they current permit will be naturally cautious . They will not want unintended consequences , and not want to be on the receiving end of complaints about an obscure service that worked yesterday but does n't today . Hence it is a much safer bet to focus the intervention on the single , well - understood , major traffic type that is driving the change . 
 Thanks everyone for supporting the idea after I answered your concerns , and helping me to get the text into a good shape . This will assist some networks to convey QUIC in an efficient way . 
 This issue is branch of where say : " ... Yes , it could be mentioned in manageability - since often QUIC packets might be rather uniform in size with occasional DLPMTUD probes ? " said : " Do we need still something for manageability ? Or is that , at least partially , already addressed by ? " 
 This is a separate note to call out that a QUIC sender can probe the path to determine whether the PMTU can be increased . When finally converges to text , we should check this is also covered . 
 This was covered in , which is now merged . 
 The manageability draft aims to provide guidance to network operators in managing QUIC traffic , yet currently says nothing about what could be a major impact for some networks . Consider a cellular / mobile network that is providing internet access to end users . Because it is an internet access service , the users have an implicit expectation that the complete range of internet packet sizes ( i.e. up to 1500 bytes ) can be exchanged . And as it is a cellular network packets are routinely encapsulated in GTP - U. The network operator also chooses to add IPSEC encapsulation in order to secure communications over a diverse range of connectivity providers . Since at least some of that connectivity is over paths with 1500 MTU , it implies that end - user datagrams over a certain size ( e.g. 1400 ) would have to be fragmented at the point of encapsulation , and reassembled before delivery to the user . Such fragmentation and reassembly comes at a cost . Importantly it implies a limited capacity for packets with sizes in the relevant range ( e.g. 1400 - 1500 ) . In a world where TCP dominates this is easily managed with MSS Clamping , so that both TCP endpoints learn the optimal size to use . In QUIC a different mechanism is required . QUIC - TRANSPORT describes how the endpoints can discover a hard path MTU limit , i.e. one where there is a clear cutoff above a certain size . I think the manageability draft needs a section that advises operators of networks where there is a soft path MTU limit . I 'll create a PR with proposed wording for this section . If anyone is unconvinced about the need for such packet size limitation , happy to discuss in more depth . 
 I agree that what you 're describing is a real problem , but given QUIC packets have the DF bit set , should n't that indicate to these networks that the QUIC packets should not be fragmented ? We have seen networks similar to what you 're describing that appear to ignore the DF bit , which I believe goes against the assumptions in 
 I 'll give two answers to this . First is the technical reason . In a cellular network this pattern can arise : ! Because the IPSEC gateway acts on a packet that is already encapsulated , it naturally looks at the lower - layer IP header rather than the user 's original IP header that had the DF bit set . Furthermore even if it performed deeper inspection and was able to take note of the DF , it would not have the ability to generate ICMP towards the server . The second is the customer expectation reason . Clearly we could configure a router upstream of the PGW to have a lower MTU interface towards the PGW . This would force the router to drop the largest packets if they had DF set . And if they did n't set it , they would be fragmented ( for v4 ) . For mainstream internet services this would be ok , and few would notice . But with a large customer base , it is extremely likely that somebody will have a service that fails to work over such a network . At the end of the day people are paying for a " normal " internet service , and at least some of them will expect 1500 byte packets to be deliverable . Would you be happy for your home internet to be one that could never support packets larger than 1400 bytes ? 
 Thanks for this explanation . If a GTP tunnel encapsulates an IP packet that has DF set , and it does not set DF in the encapsulating packet , that tunnel needs to provide the necessary mechanisms to reassemble packets that are fragmented along the tunnelled path - that adds complexity at the lower layer , but it 's allowed - providing that it does not clear DF . Some paths support an MTU larger than 1500 B , many home access networks support an MTU smaller than 1500 B. That 's why we have PMTU discovery methods . I note GTP is not alone in wishing take - on support for a larger PMTU . Many layer 2 's also transparently fragment , but this is n't something that the IETF should advocate per protocol ( even though TCP - MSS clamping works in cases for instance when IPsec is n't used ) . I do n't see any specific need for anything in QUIC , or in guidance for QUIC . This is an IP - layer decision . A tunnel / encaps needs to support the MTU that it offers . Documenting the ways in which this can happen has been a long saga in the IETF : 
  , thanks for the feedback . The tunnels draft is not that easy to follow when it gets into dense discussion . However I do like the recognition that in ICMP there 's no way to say " bigger than I 'd like , but I can deal with if needed " . That is essentially the problem here . We have a network that can handle large packets up to a certain percentage of the traffic ( because it provides the necessary mechanisms ) , but it ca n't do that for the entire web . So if I 'm understanding you correctly , a network that does not want to carry all port 443 at 1500 bytes should handle this by using PMTU discovery methods to guide senders appropriately . That is essentially what I 'm proposing in the PR . Why is this particularly , uniquely , relevant to QUIC ? Because HTTP is the bulk of internet traffic today , and clients and servers have incentives to adopt HTTP/3 . This will radically alter the present ratio of TCP : UDP , and all such networks with reassembly capacity limits will benefit from a warning that their existing TCP MSS clamping will no longer protect them . 
 From an attempt to deploy Path MTU from server to client , we saw a large number of paths where a larger MTU worked when probing , but sometime after increasing the MTU , the connection was black - holed . The solution was to switch back to the original handshake MTU after a certain period of black - holing , which avoided the connection failing . However , when this was more widely enabled , we saw : 1 ) Complaints from operators who could n't handle the excess traffic at higher MTUs 2 ) User visible issues(app - level timeouts ) on some ISPs This was the most user - visible issue I can recall our deployment having in the last few years , and it was completely unexpected , at least for me , because I was unaware of this ' can handle large packets up to a certain percentage of the traffic ' form of network operations . Additionally , it was quite difficult to link the second issue to PathMTU discovery . I can imagine a number of operator changes which would make our past PathMTU algorithm safe . One is to never fragment UDP 443 when DF is set(or it 's IPv6 ) . I 'm not sure when we 'd expect to see such changes deployed . In the meantime , I can think of a few options I do n't particularly like : 1 ) Do n't try probing up past some fixed size(1400 bytes of payload ? ) which could cause fragmentation . 2 ) Do n't try PathMTU for certain ASNs . 3 ) Adopt much more complex probing strategies that are optimized to quickly detect when a user is on one of these networks .  , do you think it 's reasonable to expect this can be changed in existing networks in 2021 or is this a multi - year process ? 
 There seem to be two questions to answer here . Q1 : Which guidance should we give to network operators in this draft ? I think it is pretty clear - if your network has one of these soft limits , you should drop oversized UDP/443 and send ICMP PTB . You will need to do this at a point in the network where you can easily send the PTB . This may be upstream of the point where fragmentation would have occurred . You raise the question of whether this should also occur if it 's ( IPv4 and DF is not set ) . I think we can leave that choice to implementations . Personally I would say do it regardless . If it 's QUIC , and DF is not set , then packets are not supposed to be larger than 1200 anyway . I would say it 's reasonable to expect that this can be changed in 2021 . Q2 : Which advice should we give to endpoints to help them avoid issues over networks that have n't implemented the guidance yet ? Perhaps before exceeding 1400 bytes of ( IP header + UDP header + QUIC ) the client could also do a conventional TCP handshake and take note of the MSS ? If it has been clamped this information could be stored and used to inform packet size decisions . Unfortunately the server does n't have this option so it comes down to your three options , or a fourth which is to only migrate to HTTP/3 over a period of weeks . During that migration look for per - ASN degradation in metrics compared to HTTP/2 . I would hope that networks that are then encountering capacity limits would be sufficiently on the ball to do something about it . If they complain to you , point them to the relevant section of the manageability draft . 
 I agree that having a relevant section would help . 
 There are a number of forums where it could be publicised , national and international . BT are plugged into many of them so I can do my bit . 
 So a few things here : QUIC is n't going to run on port 443 exclusively . Traffic might be identifiable as QUIC , but the wire image is pretty hard to distinguish . However , I think it 's OK to recommend PTB rather than fragmentation for IP packets that might be QUIC . ( Other UDP protocols need to deal with those anyway . ) I do n't think that information about the TCP MTU is transferrable to UDP , even if several other parts of the 5 - tuple are the same . Overall , I think it would be terrible if people had to have per - ASN state just to deploy the protocol . Google might be able to deploy that , but we could n't afford it . 
 Yes , accurate distinguishability of QUIC is hard , and yes you could find it on any UDP port . But imagine a case where the network operator is blissfully unaware , until critical alarms pop up warning that a particular element of the network is overloaded , and there is end - to - end measured packet loss . Reactively finding the bulk flows that are driving this is not hard , because they will be obvious . So any proactive management to avoid this occurring will rely on predictions of where QUIC will be found . The obvious one is HTTP/3 , to be found in roughly the same port number distribution as HTTP/2 ( not h2c ) , i.e. ~99 % is 443 . Others can be to this set if they are expected to become sufficiently prominent . Coming to the other points : If the received TCP MSS is not useful as a predictor for UDP MTU , then I guess that 's ruled out . I do n't think per - ASN state is a requirement for deployment . But as in any significant change , the early adopters will find the issues first . Even when being first , the speed of rollout you select will influence the size of those issues . The good news is that a gradual rollout across a range of clients and servers will collectively drive fixes across networks to support QUIC well . 
 I have n't looked whether TCP - MSS is a useful and usable hint when searching for a UDP MTU , it might be , or not - In many cases the TCP MSS is clamped ; and it can be lower than the PMTU ( even using TCP probe segments ) , so that might be an indicator ... but I also see this does not help longer term . I do agree that we need to avoid a network operator being " blissfully unaware , until critical alarms pop up " ... and this seems the correct document to call something out - and maybe the best thing might indeed be to suggest that reassembled PMTUs are only used for apps profiles that really need it ( until such apps are -eventually- updated ) or where the network processing cost is understood and does not introduce an unwanted bottleneck / blackhole . Otherwise the path ought to drop and send an ICMP PTB message - and we can note that newer mechanisms are robust to loss of this PTB . 
 I agree with  . Only deploy fragmentation and reassembly where you know it will help . Any uncertainty and the best advice is do n't . Even without DF . Ignoring DF when the type of traffic is not known seems more like malice than anything else . 
 I 'd be happy with wording in the PR that articulated a general principle that if you have fragmentation and reassembly in your network , you should look to avoid passing DF traffic through this unless you really have to . The necessary drop of DF packets should occur at a point in the network where it is possible to generate the PTB , which may be distant from the point of fragmentation . That 's the general principle and end goal . In 2021 , many network operators are likely to make only more targetted changes towards that goal , for the reasons I gave to  in . 
 “ Padding is used by QUIC to ensure ” - please add a sentence that DPLPMTUD is another reason why you might expect QUIC Padding of specific packets after connection establishment , and refer to section in QUIC transport - or people may have a surprise when they suddenly see big probe packets in their logs:- ) 
 I thought that this is what the sentence is about ? The whole sentence says : " Padding is used by QUIC to ensure that the path is capable of transferring datagrams of at least a certain size , ... " We can add a reference to the right section in the transport draft but sure we need another sentence ? However , your point about being surprised to see big packets probably rather belongs into manageability ... ? 
 Ah sorry : I was trying to highlight that there are two places where I think QUIC might be expected to use padding ... in the initial connection ... to verify the path supports the minimum size , and later whenever  is invoked , to probe to check for a larger size . Two section refs would likely be useful . ... Yes , it could be mentioned in manageability - since often QUIC packets might be rather uniform in size with occasional DLPMTUD probes ? 
 I created a PR to add references to the respective sections in the QUIC transport draft . Explaining in detail what padding is used for internally in QUIC was n't really the point here . However , it 's probably correct that the sentence should not just omit one usage ( PLDMTUD ) and reference probably do n't hurt . Do we need still something for manageability ? Or is that , at least partially , already addressed by ? 
 I think this is good to commit , with two minor issues . Thanks for the updates , this makes sense to me .
Paragraph (version A):
The spin bit provides an additional method to measure per-flow RTT
from observation points on the network path throughout the duration
of a connection.  Endpoint participation in spin bit signaling is
optional in QUIC.  That is, while its location is fixed in this
version of QUIC, an endpoint can unilaterally choose to not support
"spinning" the bit.  Use of the spin bit for RTT measurement by
devices on path is only possible when both endpoints enable it.  Some
endpoints may disable use of the spin bit by default, others only in
specific deployment scenarios, e.g. for servers and clients where the
RTT would reveal the presence of a VPN or proxy.  To avoid making
these connections identifiable based on the usage of the spin bit,
all endpoints randomly disable "spinning" for at least one eighth of
connections, even if otherwise enabled by default.  An endpoint not
participating in spin bit signaling for a given connection can use a
fixed spin value for the duration of the connection, or can set the
bit randomly on each packet sent.
Paragraph (version B):
QUIC is deliberately designed to fail rather than persist when the
network cannot support its operation.  For HTTP/3, this extends to
recommending a fallback to TCP-based versions of HTTP rather than
persisting with a QUIC connection that might be unstable.  And QUIC-
APPLICABILITY recommends TCP fallback for other protocols on the
basis that this is preferable to sudden connection errors and time
outs.  Furthermore, wide deployment of NATs with this behavior
hinders the use of QUIC's migration function, which relies on the
ability to change the connection ID any time during the lifetime of a
QUIC connection.
Paragraph (version B):
Networks with configurations that would lead to fragmentation of
large packets should drop such packets rather than fragmenting them.
Network operators who plan to implement a more selective policy may
start by focussing on QUIC.  QUIC flows cannot always be easily
distinguished from other UDP traffic, but we assume at least some
portion of QUIC traffic can be identified (see sec-identifying).  For
QUIC endpoints using DPLPMTUD it is recommended for the path to drop
a packet larger than the supported size.  A QUIC probe packet is used
to discover the PMTU.  If lost, this does not impact the flow of QUIC
data.
Paragraph (version B):
IPv4 routers generate an ICMP message when a packet is dropped
because the link MTU was exceeded.  RFC8504 specifies how an IPv6
node generates an ICMPv6 Packet Too Big message (PTB) in this case.
PMTUD relies upon an endpoint receiving such PTB messages RFC8201,
whereas DPLPMTUD does not reply upon these messages, but still can
optionally use these to improve performance Section 4.6 of DPLPMTUD.