Draft: draft-ietf-quic-recovery

Commentary: Makes PTO consistent with loss detection , which is per packet number space . This keeps a unified exponential backoff to avoid excess network load and does n't arm the PTO for ApplicationData until the handshake is complete . and  for fixing without an explicit signal Also fixes an issue where PTO was always using maxackdelay , but there is no longer a maxackdelay in the Initial and Handshake packet number spaces , so those timeouts were longer than necessary . 
 I 'd like to see the different packet number spaces operate independently because the initial handshake can use a less optimised , more complicated code path . While separate PTOs works for that , the requirement about SHOULD coalesce across packet number spaces goes against it . It requires upstream coordination and one PTO timer firing might affect the timeout of another PTO timer because coalesced packets are being sent . Overall this sounds rather complicated . 
 The description should be fixed to remove " Necessary for fixing " . That 's not true . 
 I think I 've addressed all the feedback , PTAL 
 What is the motivation behind not sending a PTO for the application data packet number space until the handshake is completed ? If I had to guess , we 're trying to not put too much load onto the network ( and thereby increase the chance that PTO packets containing Initial / Handshake data make it through ) until we know that we 're done with the handshake . However , this only works for the server side . The client completes the handshake after sending its cert and the CFIN , but it does n't know if those packets were actually received . Maybe the signal we 're actually looking for here is handshake confirmed , and not handshake completed ? 
 The key is there 's no point in arming the PTO for ApplicationData if you do n't have keys to process the ACK frame if it was received . I tried to make that clear in the text , but suggestions are welcome . 
 Thanks for the explanation . Re - reading the text , I think this makes sense now . I noticed that this also prevents the client from PTO - ing on 0 - RTT packets , which is nice . 
 Sec 5.3.1 of quic - recovery discusses how to send loss probes . Now that this can be done at any crypto level , there ought to be some suggestions ( are any MUSTs needed ? ) on what crypto level to send probes at . This is probably like the CONNECTION_CLOSE suggestion ; if you have multiple levels , it may be good to coalesce some packets . I 'm working through the logic and may have some suggestions soon . 
 There are some outstanding Issues relevant to this , including and In particular , it 's probably worth looking at PR 
 I agree , would obviate this issue . 
 In the recovery draft , when an ack - eliciting packet is sent in any packet number space , the PTO is extended . For example , if there is a long delay between a server sending Initial and Handshake , then that would delay retransmitting the Initial . A larger problem may be the delay between Initial and 0.5RTT data on the server side or Initial and 0RTT data on the client side . In both cases , prior to handshake completion , the application may write data at anytime , subject to congestion control limits and availability of ApplicationData keys . In pathological cases(ie : an application sending tiny ' keep - alive ' like pings ) the Initial of the client or server may never be retransmitted until after the connection runs out of CWND , which is likely longer than the handshake timeout . This problem was observed in gQUIC and we an extra field to track the last sent packet containing crypto data and before the handshake was complete , armed the handshake timer from that . The most straightforward fix to this I 'm aware of is , which makes PTO per PN space . 
 I did not realize that the PTO was not already expected to be per number space ! 
 Discussed in Cupertino . to rework . 
 There 's a loss pattern that QUIC can not recover from : Assume client and server perform a handshake up to the point just after the server received the ClientFinished message . No packet loss happens so far . Both peers have now completed ( but not yet confirmed ) the handshake . Right after sending the ClientFinished , the client sends a 1 - RTT packet ( e.g. containing application data ) . This packet is lost . The server sends an acknowledgement for the ClientFinished ( in a Handshake packet ) , and this packet is lost as well . Furthermore , the server also sends a 1 - RTT packet ( e.g. containing a session ticket ) . This packet is successfully received and acknowledged by the client . At this point the endpoint are in the following state : Since it received an acknowledgement for a packet sent with 1 - RTT keys , the server now confirmed the handshake , and drops the Handshake keys . On the other hand , the client is still waiting for an acknowledgement for the ClientFinished , and retransmits it . It will however never receive an acknowledgement , since the server dropped the keys already . 
 Sounds like receiving a 1 - RTT packet on the client side might need to be another way to confirm the handshake . 
 A server can send a 1 - RTT packet before even completing the handshake , so a client receiving one does n't mean anything . In this scenario , the client will continue retransmitting the last cryptographic handshake flight . That sounds terrible . But if it ever has cause to send 1 - RTT data ( as it does in Marten 's example , which I 'm not sure is relevant ) , then it will retransmit that data on PTO . When that is eventually acknowledged , that will confirm the handshake . The more troublesome scenario is where the client sends nothing , only acknowledgments . In that case the client will have a complete handshake that can not be confirmed . Would it be enough to say that " unless an endpoint sends some ack - eliciting data in 1 - RTT packets , the handshake might remain unconfirmed indefinitely " and leave it at that ? 
 Agreed . IIRC , the assumption in the design team was that the client would have something to send . +1 . We may hint that " a client MAY send a PING packet to confirm the handshake . " 
 Unfortunately , that would n't work . As long as the client has outstanding crypto data , it will only retransmit the crypto packets , see  , although I 'd prefer a SHOULD here . 
 FWIW , what I had in mind was something like " once the handshake is complete , endpoints SHOULD send ack - eliciting packets to confirm the handshake . If there is no application data to be sent , such packets can contain PING frames . " 
 No I think the state needs to progress . Resources might be tied up and timeouts might be in place to weed out bad handshakes before a longer or indefinite idle timeout . I would say the client MUST send at least one ack - eliciting 1 - RTT packet once it is able to do so , and within , say , 1 PTO . 
 If I understand the issue correctly , the problem is that server drops handshake key too early before client gets ack for its last handshake flight . Server can ack any client Short packet without client last Handshake flight . That means that sending any ack - eliciting 1RTT packet from client does not fix this issue . Coalescing missing Handshake and Short packet would suffer MTU overflow as disscused in PR . I think server has to wait for client 's confirmation of handshake before confirming handshake . It might be good idea to reconsider explicit signal again ? 
 From the : We could change this MAY to a SHOULD and it 'll resolve this issue . 
 I 'm sorry , but I 'm having trouble following the initial example . You said that the client sent a 1 - RTT packet and it got lost . In this case , it will continue to retransmit it and will eventually get an ACK , at which point the handshake will be confirmed and it will drop the keys and stop retransmitting ClientFinished . And as  indicates above , if the client is not sending at all , then it can just send ack - eliciting frames . 
 Unfortunately , it does n't . As I described in the initial example , and in , the client wo n't retransmit 1 - RTT data until the handshake is completed . 
 well in that case , the source of the problem is the retransmission algorithm , not this . Why does it make sense to not retransmit 1 - RTT data when there is an outstanding CFIN ? Just consider the simple case where the client 's second flight ( CFIN + some data ) is lost . When the PTO expires , you should resend all of that . 
 The client could have sent an arbitrary amount of 1 - RTT data , so you would n't want to retransmit all of this . As discussed on the PR ( ) , this can have some weird interaction with the congestion controller as well . 
 Sure , you would , however , want to retransmit some of it 
  , 1RTT packets can be sent on the PTO when there is unacked Handshake data , but you 're correct that if there are two MSS or more of Handshake data , the client would repeatedly retransmit that data instead of sending any 1RTT packets as written today . I think MT 's PR is heading in the right direction . do you have an alternate suggestion of how to deal with this situation ? 
 Discussed in Cupertino . With the change in and , we believe this deadlock issue is addressed . ( also , has a pending edit ) . 
 My point is 
 Has anyone thought about revisiting implicit ACK 's in handshake ? It appears to me that many problems occur because of waiting for an ACK that does n't happen ? 
 IIUC , we agreed today in Cupertino that forbidding handshake key discarding ( ) is the solution ( at least for the time being ) , based on the assumption that it works . I am starting to wonder if it works as expected . Consider the case where the client migrates to a new path once the handshake is confirmed . That is in fact the recommended behavior when server - preferred address is used . But IIUC , the existing transport logic is designed based on the assumption that path migration happens only after the exchange of Handshake packets have ceased . I think I have found at least one issue : when the client intentionally migrates to a new path while still having unacknowledged handshake data , how does it choose the new client CID when sending the Handshake packet ? Current design assumes that the server to pick the client CID from the pool of the CIDs it has received with the NEWCONNECTIONID frames . That works for 1 - RTT packets , because 1 - RTT packet does not have the Source CID field . But when using the long header packets , the sender choose the Source CID ( the source being the client in this particular case ) . What is the solution here ? 
 Regarding the way on discarding the Handshake keys , I think the failures the WG have made until now are that we have been looking for a clever method in trying to infer when a Handshake key can be dropped , and/or using 1 - RTT packet to tell the peer that the peer can drop the handshake keys ( the problem here is that the peer might continue retransmitting Handshake packets if the endpoint ceases to send 1 - RTT packets ) . Based on the discussion in Cupertino , I think that if we are to drop the Handshake keys , we should adopt a safer approach . I 'd argue that there exists a class of approaches that can be considered safe . That is to require an endpoint to retain the Handshake keys until the peer sends an explicit signal instructing the endpoint to drop the Handshake keys . Assuming that my observation in is correct , we 'd also need to adjust the transition to the handshake confirmed state to when an endpoint sends and receives that message . That signal can be a new frame , a header bit , or a KeyUpdate . 
 What if each peer sends a required PING every 1/2 RTT until it sees a 1 - RTT packet from peer or it times out using an aggressive version of PTO ? Or something ? And preferably replace handshake ACK 's with implicit ACK 's if possible . 
 Or better a each peer sends a handshake - PING ( HSPING ) such the a received HSPING indicates that the peer is still not happy . This differs from ACK 's in that it is unsolicited and therefore can not hang . EDIT : the details needs to be worked out - when to send HSPING on HS PN space and when to send on 1 - RTT PN space , and when to stop . But I think this should stop deadlocks and eventually confirm the handshake . Ordinary 1 - RTT can start before HSPINGS stop , but HS is not complete until HSPING 's are no longer required to be sent . After that point the peer can go into a mode where HS keys can be dropped , keys updated , and paths migrated . Also the connection handler can use an optimized versions for post handshake logic . 
 it sounds like you are recommending something like the previous RETIRE_KEYS proposal ? 
 I would definitely prefer something like that over never getting rid of the handshake keys . 
 I still prefer an implicit signal , and can think we can get it to work . But I would much prefer an explicit frame to keeping it forever . 
 Why not simply prohibit migration until all handshake packets have been ACKed . There is no ambiguity here , as the server is forbidden from initiating migration . 
 When the handshake is confirmed , all the Handshake data is known to be delivered and Handshake recovery state can be discarded . Previously , this was the moment Handshake keys were discarded , so the recovery doc relied on text about key discard to discard recovery state . But now that the Handshake keys are staying around longer than the Handshake confirmed point , I 'd suggest we add a paragraph to section 5.5 ( ) like : " When the handshake has been confirmed , all data sent in Handshake packet is known to be delivered and all recovery state should be discarded . " To be clear , this suggestion is not a hack . We should do this if we 're going to keep Handshake keys past Handshake Confirmed(which every solution I believe requires ? ) , regardless of how long or why . I 'll note that the server can do this upon receipt of any 1 - RTT packet , but waiting for an ACK is n't likely to delay dropping Handshake state significantly and is consistent between client and server . 
 The problem here is that even in the case where an endpoint ( e.g. , server ) has received all the handshake transcript and also the ACKs for all the ack - eliciting Handshake packets it has sent , the other side ( e.g. , client ) might have still some data outstanding ( due to ACKs being lost ) . ( Assuming that your presumption here is that the path for the Handshake packets do not change when the client migrates ) It 's not only the the receive - side state of the Handshake packets that you need to retain forever , but also the state related to that path ( e.g. , socket ) . I am not sure we 'd prefer that . This might include CIDs that are retired . 
 That might be the case ( honestly I do not remember the details of the RETIRESKEYS proposal ) , though I think I 'm inclined to suggesting using a continuous signal ( i.e. , a Key Update or burning an unused header bit ) . The reasons being i ) using a continuous signal allows us to abandon the Handshake keys earlier ii ) we do not need to define a new signal that needs to be retransmitted in the transport draft , or rely on a particular behavior defined in the recovery draft that is not generally considered as necessary for a recovery logic ( i.e. PTO " MUST " trigger a PING when there is no data to send ) . 
 I remember thinking RETIRE_KEYS was extraneous given all the implicit signals . But it seems that screwy - but - legal behavior seems to make those implicit signals risky . So let 's stop being cute and just use the signal . 
 In response to " when the client intentionally migrates to a new path while still having unacknowledged handshake data , how does it choose the new client CID when sending the Handshake packet ? " I 'm arguing the client should never have unacknowledged Handshake data when it migrates , because it waits for handshake confirmation to migrate and at that point , all Handshake data should be discarded . There is a possibility the server does not believe the handshake is confirmed but the client does . In that case , if the client did n't keep listening on the old path when it migrated AND the server did n't send anything in 1 - RTT , there would be a problem . But do n't we require both client and server to validate the path in this case , which would mean both would achieve handshake confirmation on the new path , if they had not done so previously ? 
 Yes . This is the case I am pointing out . I am not sure if there is a guarantee that handshake confirmation would be obtained before the path is validated , because the path might become validated while all the packets carrying 1 - RTT ACKs get dropped . And at that point , the endpoints might cease sending 1 - RTT packets to each other . 
 What points out is a problem . Keeping HS keys forever prevents the deadlock , but it does so by admitting that HS packets can be sent forever . This means after migration as well . That is something we do n't consider in the spec . I do n't think you 're right about this : Handshake data is discarded when the handshake keys are dropped , not when confirmation is reached ( see . Now with handshake keys forever , the client can keep retransmitting forever . And as notes , the server can surely send keep retransmitting as well since if it has n't reached confirmation yet . There are a few ways to curb this ... but I 'm strongly inclined towards an explicit signal from both sides indicating handshake confirmation , after which migration can be initiated . 
 There is a much larger issue at stake here . The currently published proofs for the TLS 1.3 handshake do not allow the server to process 1 - RTT packets until the client finished has been verified . It does not mean there is an attack , because of the defensive design of TLS 1.3 which implicitly authenticates the transcript in the transport secret derivation . However , this is still a weaker argument than checking the finish , particularly with respect to downgrade attacks . If a server supports a transport ciphersuite with broken integrity ( e.g. someone discovers a way to forge CCM8 tags ) , then clients of this server that support strong ciphersuites may still be attacked . There is also a clear risk with client authentication . If a server is configured to require a client certificate , but QUIC gives to the application 1 - RTT packets before the client finished flight has been received , the application may incorrectly believe that this data has passed the client authentication check . A reliable way to prevent such issues is to mandate that every 1 - RTT packet sent by the client before the client finished frame is acknowledged by the server must include in the same datagram a copy of the finished frame ( in the form of a separate handshake packet , before the short 1 - RTT packet ) . The overhead is limited because finished messages are short , so even with the overhead of the long packet headers and AEAD tag , less than 100 bytes are used in the datagram ( leaving the rest for the 1 - RTT packet ) . If this is done , it is possible to enforce that a server must have checked the client finished before processing 1 - RTT packets from the client . Adding some separate signal for handshake confirmation is a terrible idea - in TLS , receiving the peer 's finished message IS the the explicit handshake confirmation signal ( which is why CCS is removed in 1.3 ) . Restoring a transport - level signal is a clear step backwards and must be avoided . 
 Hi since you 're describing a " much larger issue " that I agree should be discussed , could you please open a separate issue ? This issue is about a specific deadlock that can happen during the handshake , not about the security properties of the handshake . 
 This was closed erroneously ... apparently we 're not quite fully zen with  's AI yet . 
 Discussed in Singapore ; CfC . 
 LGTM
Paragraph (version A):
The PTO period is the amount of time that a sender ought to wait for
an acknowledgement of a sent packet.  This time period includes the
estimated network roundtrip-time (smoothed_rtt), the variance in the
estimate (4*rttvar), and max_ack_delay, to account for the maximum
time by which a receiver might delay sending an acknowledgement.
Paragraph (version A):
When a PTO timer expires, the PTO period MUST be set to twice its
current value.  This exponential reduction in the sender's rate is
important because the PTOs might be caused by loss of packets or
acknowledgements due to severe congestion.  The life of a connection
that is experiencing consecutive PTOs is limited by the endpoint's
idle timeout.
Paragraph (version A):
A sender computes its PTO timer every time an ack-eliciting packet is
sent.  A sender might choose to optimize this by setting the timer
fewer times if it knows that more ack-eliciting packets will be sent
within a short period of time.
Paragraph (version B):
The PTO period is the amount of time that a sender ought to wait for
an acknowledgement of a sent packet.  This time period includes the
estimated network roundtrip-time (smoothed_rtt), the variance in the
estimate (4*rttvar), and max_ack_delay, to account for the maximum
time by which a receiver might delay sending an acknowledgement.
When the PTO is armed for Initial or Handshake packet number spaces,
the max_ack_delay is 0, as specified in 13.2.5 of QUIC-TRANSPORT.
Paragraph (version B):
When a PTO timer expires, the PTO period MUST be set to twice its
current value.  This exponential reduction in the sender's rate is
important because the PTOs might be caused by loss of packets or
acknowledgements due to severe congestion.  Even when there are ack-
eliciting packets in-flight in multiple packet number spaces, the
exponential increase in probe timeout occurs across all spaces to
prevent excess load on the network.  For example, a timeout in the
Initial packet number space doubles the length of the timeout in the
Handshake packet number space.
Paragraph (version B):
The life of a connection that is experiencing consecutive PTOs is
limited by the endpoint's idle timeout.
Paragraph (version A):
When a PTO timer expires, a sender MUST send at least one ack-
eliciting packet as a probe, unless there is no data available to
send.  An endpoint MAY send up to two full-sized datagrams containing
ack-eliciting packets, to avoid an expensive consecutive PTO
expiration due to a single lost datagram.
Paragraph (version A):
Probe packets sent on a PTO MUST be ack-eliciting.  A probe packet
SHOULD carry new data when possible.  A probe packet MAY carry
retransmitted unacknowledged data when new data is unavailable, when
flow control does not permit new data to be sent, or to
opportunistically reduce loss recovery delay.  Implementations MAY
use alternate strategies for determining the content of probe
packets, including sending new or retransmitted data based on the
application's priorities.
Paragraph (version A):
Bursting more than an initial window's worth of data into the network
might cause short-term congestion and losses.  Implemementations
SHOULD either use pacing or reduce their congestion window to limit
such bursts.
Paragraph (version B):
A sender MAY implement alternative mechanisms to update its
congestion window after periods of under-utilization, such as those
proposed for TCP in RFC7661.