Draft: draft-ietf-quic-multipath

Commentary: This derives from the discussions on issue 
 The commit applies the changes suggested in the previous reviews . With those changes , I think we have a good basis for discussing a unified solution in the working group . 
 Thanks for the review . I have applied the editorial changes . You are asking for something more regarding the " specific logic " required when supporting zero - length CID . There are in fact two pieces to that : logic at the receiver , i.e. , the node that chose to receive packets with zero - length CID ; and logic at the sender , i.e. , the node that accepts to send data on multiple paths towards a node that uses zero - length CID . This is explained in details in the section " Using Zero - Length connection ID " , so I guess what we need is a reference to that section . 
 Comment by ( on PR ): I think we expand this section a bit as we found getting the delay right was non - trivial . QUIC time - stamp would work for sure . But if one decides not to use QUIC timestamp , we probably want to explain what is needed to be done here . 
 I have submitted PR for this one . 
 Closing as got merged . 
 In section 3.1 , we wrote When the multipath option is negotiated , clients that want to use an additional path MUST first initiate the Address Validation procedure with PATHCHALLENGE and PATHRESPONSE frames described in of ] . After receiving packets from the client on the new paths , the servers MAY in turn attempt to validate these paths using the same mechanisms . With multipath , we can not simply write that the server MAY validate the new path . There is a risk of a malicious client that operates as follows : client creates path from its own address ,   is validate by server client starts long download from server client creates a second path to server using a spoofed address , B client starts address validation for B but does not care about the response from the server client stops acknowledging packets over the first path to force the server to move packets to the second one and then opportunistically acknowledges packets that should have been sent by the server over that second packet ( SPNS would probably make such an attack simpler than MPNS ) We probably need to start working on the security considerations section of the draft . 
 This is already covered in the QUIC threat model . The server is supposed to validate new paths by sending a " path challenge " on that path , then waiting for a challenge response . If the client spoofs a path , it will never receive the challenge , and thus the server will never receive the response and will not validate the path . See section 8.2 of RFC 9000 , path validation . Security section 21.1.3 .   states that " Path validation ( ) establishes that a peer is both willing and able to receive packets sent on a particular path . " 
 So yes , the current text should be " the servers MUST in turn attempt to validate these paths using the same mechanisms , as specified in { { Section 8.2 of RFC9000 } } . " 
 No , this should not be a MUST . It is true that the path MUST be validated before it can be used . However , just because the serve receives a new packet on a path , that does n't automatically mean that the server has to start path validation . It could e.g. also decide to simply withdraw that packet . 
 Actually , this is also not right . This is what RFC9000 says ( section 9 ): So I guess we should maybe not use normative language here but refer to section 9 in RFC9000 instead ... ? 
 Just submitted PR for this issue . 
 This draft initially originates from a merging effort of previous  proposals . While many points were found to be common between them , there remains one design point that still requires consensus : the number of packet number spaces that a Multipath QUIC connection should support ( i.e. , one for the whole connection vs. one per path ) . The current draft enables experimentation with both variants , but in the final version we will certainly need to choose between one of the versions . 
 The main issues mentioned so far : Efficiency . One number space per path should be more efficient , because implementations can directly reuse the loss - recovery logic specified for QUIC ACK size . Single space leads to lots of out of order delivery , which causes ACK sizes to grow Simplicity of code . Single space can be implemented without adding mane new code paths to uni - path QUIC  header . Single space works well with NULL length CID The picoquic implementation shows that " efficiency " and " ack size " issues of single space implementations can be mitigated . However , that required significant improvements in the code : to obtain good loss efficiency , picoquic remembers not just the path on which a packet was sent , but also the order of the packet on this path , i.e. a " virtual sequence number " . The loss detection logic then operates on that virtual sequence number . to contain ACK size , picoquic implements a prioritization logic to select the most important ranges in an ACK , avoid acking the same range more than 3 or 4 times , and keep knowledge of already acknowledged ranges so range coalescing works . I think these improvements are good in general , and I will keep them in the implementations whether we go for single space or not . The virtual sequence number is for example useful if the CID changes for reasons not related to path changes in multiple number space variants . It is also useful in unipath variants to avoid interference between sequence numbers used in probes and the RACK logic . The ACK size improvements do reduce the size of ACKs in presence of out of order delivery , e.g. , if the network is doing some kind of internal load balancing . On the other hand , the improvements are somewhat complex , would need to be described in separate drafts , and pretty much contradicts the " simplicity of code " argument . So we are left with the " Null length CID " issue . I see for cases : Client CID Priority ----------------------- long Used by many implementations NULL Preferred configuration of many big deployments long Rarely used , server load balancing does not work NULL Only mentioned in some P2P deployments The point here is that it is somewhat hard to deploy a large server with NULL CID and use server load balancing . This configuration is mostly favored by planned P2P deployments . 
 The big debate is for the configuration with NULL CID on client , long CID on server . The packets from server to client do not carry a CID , and only the last bytes of the sequence number . The client will need some logic to infer the full sequence number before decrypting the packet . The client could maybe use the incoming 5 tuple as part of the logic , but it is not obvious . It is much simpler to assume a direct map from destination CID to number space . That means , if a peer uses a NULL CID , all packets sent to that peer are in the same number space . 
 Revised table : Client CID What -- -- long Multiple number space NULL Multiple number spaces on client side ( one per CID ) , single space on server side long Multiple number spaces on server side ( one per CID ) , single space on client side NULL single number space on each side If a node advertises both NULL CID and multipath support , they SHOULD have logic to contain the size of ACK . If a node engages in multipath with a NULL CID peer , they SHOULD have special logic to make loss recovery work well . 
 I think the above points to a possible " unified solution " : if multipath negotiated , tie number spaces to destination CID , use default ( N=0 ) if NULL CID . use multipath ack ; in multipath ACK , identify number space by sequence number of DCID in received packets . Default that number to zero if received with NULL CID . in ABANDON path , identify path either by DCID of received packets , SCID of sent packet , or " this path " ( sending path ) if CID is NULL . 
 I like the proposal of the " unified solution " . I think the elegance lies in the fact that it allows us to automatically cover all four cases listed above . The previous dilemma for me was that on one hand we have some use cases where we need to support more than two paths and separate PN makes the job easier , but on the other hand , I think we should not ignore the NULL CID use cases as it is also important . Now , with this proposal , a big part of the problem is solved . The rest of challenge is to make sure single PN remains efficient in terms of ACK and loss recovery . On that part , we plan to do an A / B test and would love to share the results when we get them . There is one more problem as pointed in issue , when we want to take hardware offloads into account . In such a case , we may still need single PN for long server CID . However , if hardware supports nonce modification , this problem can be addressed with the proposed " unified solution " . 
 If the APi does not support 96 bit sequence numbers , it should always be possible to create an encryption context per number space , using Key = current key and ID = current - ID + CID sequence number . Of course , that forces creation of multiple context , and that makes key rotation a bit harder to manage . But still , that should be possible . 
 Thanks for the summary I think one point is missing in your list which is related to issue . Use of a single packet number space might not support ECN . Regarding the unified solution : I think what you actually say is that we would specify both solutions and everybody would need to implement both logics . At least regarding the " simplicity of code " argument , that would be the worst choice . If we can make the multiple packet number spaces solution work with one - sided CIDs , I 'm tending toward such an approach . Use of multiple packet number spaces avoids ambiguity / needed " smartness " in ACK handling and packet scheduling which , as you say above , can make the implementation more complex and , moreover , wrong decisions may have large impact on efficient ( both local processing and on - the - wire ) . I do n't think we want to leave these things to each implementation individually . 
 The summary Christian made above about design comparison sounds indeed quite accurate . Besides ECN , my other concern about single packet number is that it require cleverness from the receiver side if you want to be performant in some scenarios . At the sender - side , you need to consider a path - relative packet number threshold instead of an absolute one to avoid spurious losses . Just a point I think we did not mentioned yet is that there can be some interactions between Multipath out - of - order number delivery and incoming packet number duplicate detection . This requires maintaining some state at the receiver side , as described by With single packet number space , the receiver should take extra care when updating the " minimum packet number below which all packets are immediately dropped " . Otherwise , in presence of paths with very different latencies , the receiver might end up discarding packets from a ( slower ) path . I 'm also preferring the multiple packet number spaces solution for the above reasons . I 'm not against thinking for a " unified " solution ( the proposal sounds interesting ) , but I wonder how much complexity this would add compared to requiring end hosts to use one - byte CIDs . 
 I think the issue is not really so much " single vs multiple number space " as " support for multipath and NULL CID " . As noted by  , there is an implementation cost there . My take is : If a node uses NULL CID and proposes support of multipath , then that node MUST implement code to deal with size of acknowledgements . If a node sees a peer using NULL CID and supporting multipath , then that node MUST either use only one path at a time or implement code to deal with impact on loss detection due to out of order arrivals , and impact on congestion control including ambiguity of ECN signals . Then add sections on what it means to deal with the side of acknowledgements , out of order arrivals , and congestion control . I think this approach ends up with the correct compromises : It keeps the main case simple . Nodes that rely on multipath just use long CID and one number space per CID . Nodes that use long CID never have to implement the ACK length minimization algorithms . Nodes that see a peer using NULL CID have a variety of implementation choices , e.g. , sending on just one path at a time , sending on mostly one path at a time in " make or break " transitions , or implementing sophisticated algorithms . If we do that , we can get rid of the single vs multiple space discussion , and end up with a single solution addressing all use cases . 
 Looking at all the discussions here and in other issues such as ECN , I think that we should try to write two different versions of section 7 : one covering all the aspects of single packet number space ( RTT estimation , retransmissions , ECN , handling of CIDs , ... ) one covering all the aspects of multiple packet number space ( RTT estimation , retransmissions , ECN , handling of CIDs , ... ) There would be some overlap between these two sections and also some differences that would become clear as we specify them . At the end of this writing , we 'll know whether it is possible to support / unify both or we need to recommend a single one . The other parts of the document are almost independent of that and can evolve in parallel with these two sections . However , I do n't think that such a change would be possible by Monday 
 I do n't think we should rush changes before we have agreed on the final vision . 
 It might be helpful to have these options as PRs ( without merging ) them , so people can understand all details of each approach . 
 I agree with that supporting multi - path with null CIDs is a more fundamental issue than the efficiency comparison between single PN and separate PN , as it would ultimately impact the application scope of multipath QUIC . But indeed , we might want to implement proposed solution first and then decide if we want to adopt such a unified approach . 
 Since prodded me , we now have a PR for the " unified " proposal . 
 I totally agree with Christian that the issue is about " support for multipath and NULL CID " , and the solution that Christian suggested looks really great ! It both takes advantage of multiple spaces , and support NULL CID users without affect the efficiency of ACK arrangements . Besides , the solution is more convenient for implementations , because If both endpoints uses non - zero length cids , endpoints only need to support multiple spaces , and if one of the endpoints use NULL CID , it could use single pn space in one direction and could support NULL CID and multipath at the same time . 
 Very high level summary of IETF-113 discussion seems that there is interest and likely support for the unified solution ( review minutes for further details ) .
As currently defined in this version of the draft the use of multiple
packet number spaces requires the use of connection IDs is both
directions.  Today's deployments often only use destination
connection ID when sending packets from the client to the server as
this addresses the most important use cases for migration, like NAT
rebinding or mobility events.  Further discussion and work is
required to evaluate if the use of multiple packet number spaces
could be supported as well when the connection ID is only present in
one direction.
This extension uses multiple packet number spaces.  When multipath is
negotiated, each destination connection ID is linked to a separate
packet number space.  Using multiple packet number spaces enables
direct use of the loss recovery and congestion control mechanisms
defined in QUIC-RECOVERY.
If for any one of the endpoints the parameter is absent or set to 0,
or if the two endpoints select incompatible values, one proposing 0x1
and the other proposing 0x2, the endpoints MUST fallback to QUIC-
TRANSPORT with single path and MUST NOT use any frame or mechanism
defined in this document.
If an endpoint proposes the value 0x3, the value proposed by the
other is accepted.  If both endpoints propose the value 0x3, the
value 0x2 is negotiated.
If for any one of the endpoints the parameter is absent or set to 0,
the endpoints MUST fallback to QUIC-TRANSPORT with single active path
and MUST NOT use any frame or mechanism defined in this document.
Endpoints negotiate the use of one packet number space for all paths
or separate packet number spaces per path during the connection
handshake nego.  While separate packet number spaces allow for more
efficient ACK encoding, especially when paths have highly different
latencies, this approach requires the use of a connection ID.
Therefore use of a single number space can be beneficial when
endpoints use zero-length connection ID for less overhead.
If the multipath option is negotiated to use one packet number space
for all paths, the packet sequence numbers are allocated from the
common number space, so that, for example, packet number N could be
sent on one path and packet number N+1 on another.
ACK frames report the numbers of packets that have been received so
far, regardless of the path on which they have been received.  That
means the senders needs to maintain an association between sent
packet numbers and the path over which these packets were sent.  This
is necessary to implement per path congestion control.
When a packet is acknowledged, the state of the congestion control
MUST be updated for the path where the acknowledged packet was
originally sent.  The RTT is calculated based on the delay between
the transmission of that packet and its first acknowledgement (see
compute-rtt) and is used to update the RTT statistics for the sending
path.
Also loss detection MUST be adapted to allow for different RTTs on
different paths.  For example, timer computations should take into
account the RTT of the path on which a packet was sent.  Detections
based on packet numbers shall compare a given packet number to the
highest packet number received for that path.
In this case, ACK frames report the numbers of packets that have been
received so far, regardless of the path on which they have been
received.  That means the senders needs to maintain an association
between sent packet numbers and the path over which these packets
were sent.  This is necessary to implement per path congestion
control, as explained in zero-length-cid-loss-and-congestion.
Loss detection as specified in QUIC-RECOVERY uses algorithms based on
timers and on sequence numbers.  When packets are sent over multiple
paths, loss detection must be adapted to allow for different RTTs on
different paths.  When sending to zero-length CID receivers, packets
sent on different paths may be received out of order.  Therefore
senders cannot directly use the packet sequence numbers to compute
the Packet Thresholds defined in Section 6.1.1 of QUIC-RECOVERY.
Relying only on Time Thresholds produces correct results, but is
somewhat suboptimal.  Some implementations have been getting good
results by not just remembering the path over which a packet was
sent, but also maintaining an order list of packets sent on each
path.  That ordered list can then be used to compute acknowledgement
gaps per path in Packet Threshold tests.
If the multipath option is enabled with a value of 2, each path has
its own packet number space for transmitting 1-RTT packets and a new
ACK frame format is used as specified in ack-mp-frame.  Compared to
the QUIC version 1 ACK frame, the ACK_MP frames additionally contains
a Packet Number Space Identifier (PN Space ID).  The PN Space ID used
to distinguish packet number spaces for different paths and is simply
derived from the sequence number of Destination Connection ID.
Therefore, the packet number space for 1-RTT packets can be
identified based on the Destination Connection ID in each packets.
If packets contain a non-zero CID, each path has its own packet
number space for transmitting 1-RTT packets and a new ACK frame
format is used as specified in ack-mp-frame.  Compared to the QUIC
version 1 ACK frame, the ACK_MP frames additionally contains a Packet
Number Space Identifier (PN Space ID).  The PN Space ID used to
distinguish packet number spaces for different paths and is simply
derived from the sequence number of Destination Connection ID.
Therefore, the packet number space for 1-RTT packets can be
identified based on the Destination Connection ID in each packets.