Change:
As pointed out in , the drafts assume that the traffic keys are generated synchronously. But that's not what we see in the wild. In fact, clients often need to spend time while verifying the certificate, and some if not most of the clients do receive and send QUIC packets while they do so. This PR acknowledges the existence of such implementation, recommending them to buffer the undecryptable packets while they calculate the next key, to prevent loss events that would kill the slow start. It also allows endpoints to postpone arming the PTO timer when it knows that the peer is going to send a handshake message.
NAME Thank you for your comments. I think the two new commits address your concerns. PTAL.
NAME Thank you for your help in moving this PR forward. I'm a bit hesitant to call this PR editorial though, previously endpoints were expected to process TLS handshake messages synchronously, now we have a MAY that allows them to process them in an asynchronous manner (while waiting for TLS to provide new keys).
To make the change of status clear, the change to probing keeps probes for application data disabled until handshake confirmation rather than handshake completion. This is based on the conclusions in . I think that PR is an incremental change in that direction, but more is needed to complete the set of changes proposed there. My suggestion would be to get agreement on the overall strategy, as proposed. Then we can take this PR and follow up with more comprehensive/systemic fixes.
The text here (aside from needing a rebase) is OK for TLS. It's the asynchronous certificate validation stuff that bothers me and that can be done as the next step (a step for which it would be good to have this PR merged).
Let's not duplicate existing requirements on ACK behavior in recovery, given we decided to move it to transport and that text is still getting cleaned up.
When reported ACK Delay would cause the RTT sample to go below minrtt, we ignore the ACK Delay. In general, I think that this provides the right incentives to report delays accurately. If the true minimum RTT is lower than the observed value, you add a few samples that are inflated by up to maxackdelay. URL the handshake, where we don't have an effective cap on ACK Delay, so we effectively have unbound inflation. And minrtt is likely not well established. Should we discard these samples instead?
I think this is a potential issue, but I'd like to see this addressed in with some text to advise this is a potential issue that MAY be solved by ignoring RTT samples.
Happy to have this rolled into as long as we can do so expediently. I don't want a design issue floating around after this next batch of consensus calls. NAME do you think that something could be done?
I tend to think that if we are going to have some advice, it would be something like "MAY ignore the first ACK sample for Handshake and ApplicationData packet number spaces." IIRC, NAME suggested something like that somewhere. Though, if we are going to have something like that, I think we also need to state that using a bad RTT sample might still better than relying on initial RTT of 333ms.
I've added some text in (lines 351 - 356) that should capture this point.
Thanks NAME that new text works for me.
I'm splitting this from () because the set of changes are disjoint from the current PR for that issue and I think having both issues in one is making it harder to follow. (The rest of this text is an edit of my last post on that issue). The issue is that ACKs for Handshake and 1-RTT packets could be delayed because the peer doesn't have the keys(either because it takes time to derive them, or it's missing crypto data to derive the next key). The first RTT sample in Handshake and 1-RTT packet spaces could have substantial delay in excess of the 'no ack delay' in Handshake or maxackdelay for 1-RTT, which inflates SRTT and RTTVar. If there's no loss, this fixes itself without harm, which is nice. And it's not as bad for ApplicationData, since that will have a non-0 maxackdelay, which may be enough to absorb the endpoint delay generating keys or doing cert verification, though it won't typically be enough to adjust for a packet being lost and the data being PTO'd and then that making keys available. In , Jana's suggested changing ackdelay to including the time packets were buffered as well as and ignoring maxackdelay for the first sample in a PN space. I think this is a sensible mitigation, though it doesn't guarantee the issue will not occur. If no Initial ACKs have been received and the ACK with the large ackdelay is the first ACK of the connection, then you still can't use ack_delay. The one change we've found mitigates this issue entirely is coalescing packets from the next PN space(Handshake or 1RTT) and then processing the full coalesced packet before processing all buffered packets and sending an ACK. That ensures there's one new packet being acknowledged and you get a non-delayed RTT sample. It doesn't seem onerous to add a "SHOULD process all packets in the current datagram and all buffered packets before sending an ACK to avoid generating multiple inflated RTT samples" to me?
Repeating this here, so that the PR comments don't get lost: This is about a sender having to buffer packets that might contain ACK frames. Usually, this is because the sender doesn't have the keys necessary, so they hold the packets until they do. When processing the ACK frame, some time might have passed. So we now have 5 times in play: when the largest acknowledged packet was a) sent, and b) received; and when the packet containing the ACK was c) sent, d) buffered, and e) processed. The critical difference here is that we now recognize the intentional delays at the sender in addition to intentional delays at the receiver. The receiver calculates and transmit and sends that in ACK Delay. We have a commitment that . That doesn't need to change. The spec currently assumes that the sender calculates RTT as (or to include the cap on receiver delay). That is, the time between when the ACK is processed and when the largest acknowledged was sent, less the ACK Delay. But that's wrong. It's that we care about. So all we need to say is that if the sender buffers packets for processing, then it needs to calculate RTT based on the time that the packet entered the buffer. That is, we capture any intentional delays added by the sender. Then we don't have to worry about the maxackdelay cap, because receiver behaviour always follows the contract.
Sorry, I should correct that last bit: We only have to worry about the maxackdelay cap when the packet that is being acknowledged might have been buffered. Thus, it makes sense to remove the cap for packets that might have been buffered (i.e., those that are sent prior to handshake confirmation).
Also from the PR. I had never considered the case when the sender has buffered a packet that contains an ACK frame until you pointed it out. The PR was intended to deal with the case when a peer receives packets it can't immediately decrypt and acknowledge. But I added a sentence about subtracting the amount of time buffered when processing a buffered packet containing an ACK. It can only happen for a server that hasn't reached handshake confirmed, so it's fairly rare.
It seems to me that this issue is an alternative resolution to one specific part of what is being proposed as the resolution to . Specifically, quoting from URL, the approach proposed here is an alternative to the line being emphasized below. 1-RTT data until the handshake is complete (Section 6.2.1). This is not quite adequate, since it still allows a client to arm a PTO timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that the server has received the CFIN. The proposed solution here is to change this condition from "handshake complete" to "handshake confirmed". Meaning that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows that the peer has the keys necessary to read and write in 1-RTT, and can therefore respond with an ACK. missing subtle deadlock issues. the data appropriately. ackdelay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. For this, we need the following: and the ack being buffered) confirmation. Those are the ones that might have been buffered at the peer until keys were ready. 1-RTT packets, instead of dropping them, as dropping packets has significant impact on congestion control. Note that this matches what those implementations already do. The proposed resolution in recommends the sender to retain one bit of constant for each packet it sends, and use it to see if ackdelay should be capped. The resolution proposed here is to retain a mutable bit for each packet number space, and flip that bit when the seder receives the first ack. Until that bit is flipped, the sender does not cap ack_delay. As correctly pointed out in the , this approach relies on the receiver processing all the buffered packets at once then sending an ACK. I think that both approaches work, and do not have a strong preference among the two. I'd also note both approaches would work correctly once we adopt the basis of the latter approach; i.e. recommend the receiver to process all buffered packets before sending an ACK. Therefore, it seems to me that the only question is if we want to have that recommendation. If the answer is yes, then both approaches would work, and the choice between the approaches becomes an editorial preference (i.e. write down either or both).
I actually updated the PR to align more closely with the original proposal after NAME pointed out that they weren't identical if all buffered packets weren't processed at once(which is something I'd implicitly assumed). I could go either way on that, but I do think it's easier to only special case the first ACK for a PN space rather than store a bit per packet. It also mitigates the issue for those that don't follow this new MAY. However, one difference from the above is this doesn't rely on handshake confirmation, since this is just as much a problem for Handshake packets as it is 1-RTT packets, so I think we need to use a more general statement about the sender not knowing if the receiver has the keys to process the packets.
NAME (swapping paragraphs being quoted) I am happy to see the issue addressed for all epochs if that's the preference of others. FWIW I tend to think that we should use SHOULD, especially if we are going to also cover the delay of Handshake ACKs. Max-ack-delay of Handshake packets are assumed to be zero. Therefore, we would see the problem on any environment where the time required for processing a TLS handshake message is relatively greater than the path delay. Including datacenter networks where RTT is small, or slow IoT devices.
Let's mark this as design and run the process.
During the handshake, there are moments when an endpoint receives a packet from its peer and calculates the traffic keys of the next PN space. To give some examples, a server receives an Initial packet containing ClientHello, and derives the Handshake keys. Or, a client receives the Handshake packets, verifies the handshake transcript (including the certificate chain) and derives the 1-RTT keys. While deriving the traffic keys of the next PN space, an endpoint might receive packets belonging to the next PN space. Note that the process of deriving the traffic keys might take some time. I see Chrome running on a PC spending ~100ms for verifying the certificate chain, when it is launched from the command line along with the URL to be fetched. I presume that such launch-to-fetch-URL behavior would not be uncommon on PCs and handheld devices. Another (more) extreme case would be when OCSP is involved. In these cases, the time required for deriving the new keys could take more than a handful of RTTs. Hence the following questions, that might be worth clarifying in the specifications: Q1. How should an endpoint behave when it takes time to derive the traffic keys of the next PN space? Q1.1. Should the endpoint continue ACKing packets of the current PN space? Q1.2. Can the endpoint discard packets of the next PN space, as it can expect those packets to be retransmitted? Q2. How should the peer behave? Re Q1.1, I think that the answer is yes, it should. Such an endpoint should calculate the next set of traffic keys asynchronously. It should continue running the QUIC transport for the existing PN space. To give an example, while the client verifies the certificate chain, it should ACK and continue ACKing the Handshake packets that it receives. Re Q1.2, I think the answer is no. An endpoint SHOULD buffer packets of the next PN space while it calculates the traffic keys for that PN space. This is because the peer cannot distinguish the endpoint dropping packets vs. packets being dropped by the network. Dropping the packets lead to the peer's congestion controller exiting the startup phase. That is going to have a negative effect on performance. Re Q2, I would argue that, ideally, the peer SHOULD refrain from arming the PTO timer for the packets of the next PN space, when it knows that the endpoint has all the information to derive the traffic keys of the next PN space, and knows that the endpoint would send something once those traffic keys are derived. To give an example, a server SHOULD NOT arm the PTO timer for 0.5 RTT data until it receives ClientFinished. At the very least, we should caution the server to not send Handshake(PING) packets alongside the 1-RTT probe, as ACKs for Handshake(PING) packets would reset the PTO timer.
We encountered a problem similar to the one described here in our stack. The latter part of a coalesced packet was being dropped because we had to spend time validating the certificate. We ultimately added a small amount of buffering to address this. And I have debugged a case where handshakes were failing due to the interaction between PTO timers in different packet number spaces. For that, I don't have a good solution yet; I'm not convinced that deferring PTO is the right answer. But I don't think that this requires anything in specifications, aside from perhaps a caution, noting that we already have a caution in . That already suggests and permits (though doesn't recommend) buffering. We could add a note that failing to buffer legitimate packets could lead to marking delivered packets as lost unnecessarily and performance degradation
I would argue that a probe packet is expected to be sent when an endpoint does not receive a response that it expects. That's necessary because it is the only way to guarantee forward progress (and therefore ). But in this particular case that we are discussing, the ball is on the side of the peer. The peer is expected to respond with a new TLS handshake message contained by a QUIC packet using the next PN space. To paraphrase, sending packets before knowing that the peer is ready to decrypt them is a performance optimization. Doing that once is fine, but having a MUST for repeating that is questionable. While I am happy to discuss the nuance, I do think that it needs to be something other than "MUST send a probe." Note also that an endpoint sending PTOs in this case would in turn require the peer to have a larger buffer for storing undecryptable packets in order to avoid buffer overflow (that leads to performance degradation). Aside from relaxing the MUST pointed out above, I think I'd be fine with just adding notes. Though I might argue that it is more important than the existing note that discusses about an "exceptional" case; some endpoints would almost always need time to calculate the next keys.
FWIW, states that: as keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption level are available. These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. Part of the problem is that this is untrue. TLS stacks sometimes take time in providing the net set of traffic keys. And QUIC stacks do not even pretend that TLS stacks are acting in a synchronous manner. Instead, they intentionally send ACKs for Handshake packets that are received later than the 1-RTT packets they buffer. This is a behavior that is observable from the peer, and leads to the exit of slow start depending on how you design the loss recovery. I think we might want to use normative words stating what the expectations are.
Thanks for filing this. I think I mostly understand the issue, and I think this is mostly a correctness issue with Chrome. That being said, we may want to add some caution that this is a potential issue and to avoid handshake operations(ie: cert verification on the client and remote handshaking on the server) from excessively delaying acknowledgements. I've seen lots of cases when the cert is spread over 3 packets and Chrome ACKs the first two packets before the cert is verified, so this doesn't seem to be a case of key availability. Google servers don't have cert compression enabled yet, so many certs don't fit into 1 or 2 packets. The recovery draft already says that peers don't arm for ApplicationData(including 0.5RTT) before the handshake is confirmed. Knowing the peer has the information to derive the keys isn't really enough, since you'd also have to know the peer has received a packet of the next encryption level. As a result, I can't think of any changes to this part of the PTO logic I'd want to make.
NAME I agree that the there is no design changes necessary to the recovery draft. The crux of the problem is the TLS draft. It currently states: However, contrary to what the TLS draft says, Chrome (and IIUC what NAME says, Firefox also) processes the TLS handshake messages containing the certificate chain in an asynchronous manner. Consider the following example, where a web browser receives four packets: Handshake (PN=1, EncryptedExtensions) Handshake (PN=2, Certificate, CertificateVerify, Finished) 1-RTT (PN=3, NewSessionTicket) (30ms later) receives Handshake (PN=4, PTO) The TLS draft suggests that client would be processing PN=3 before PN=4, because the browser would process the TLS handshake transcript synchronously and therefore it would have the 1-RTT keys when it recognizes PN=3 for the first time. But the reality is that, if the browser takes 100ms to verify the certificate chain, the client buffers PN=3, acking PN=4 before PN=3, even though the interval between PN=3 and PN=4 is greater than maxackdelay. As can be seen, this is an observable behavior from the server-side. Therefore, we need to fix the TLS draft at least (assuming that we agree that browsers are doing the correct thing).
One more note is that the RTT samples can be greatly inflated if a packet(ie: the Initial) is lost and the PTO fires to retransmit the data, and that causes the previously buffered Handshake packets to be acknowledged. Coalescing an ack-eliciting Handshake packet with the Initial typically mitigates this, as long as it's processed before the buffered packets. Since coalescing is already an optimization in the draft, it may be worth a note that it can prevent inflated RTT samples. I considered an algorithm to discard these RTT samples, but filtering RTT samples is a dark art I'd rather not put into the recovery spec unless it's absolutely necessary. We've observed some cases where this causes the handshake to timeout because the server's PTO is so long, and our handshake timeout is fairly short(9 or 10 seconds).
The discussion about RTT estimate is interesting. I'm not sure about the Initial vs. Handshake case, but when the client spends X milliseconds (e.g. X=100) verifying the certificate chain at the same time buffering 0.5 RTT data it receives, then the server would obtain an RTT sample of milliseconds (e.g., 75 ms). That's not great. One way of resolving the issue would be to not cap ackdelay using maxackdelay for ApplicationData packets that were sent during the handshake. The rationale for not activating PTO for 1-RTT packets during the handshake is that the peer might not immediately have the necessary keys to process those packets. We assume the peer to buffer such packets. When the sender has such an assumption, it does not really make sense to cap the ackdelay. I think doing something like that would be fine, even though I would not push on recommending such a strategy.
NAME raises a good point about inflated RTTs. NAME point about maxackdelay actually exposes a grey area in the current semantics of that parameter. It is expected to be the maximum delay that the endpoint introduces. Other delay is supposed to be considered to be part of the "path". This is a situation where that assumption doesn't hold well. Separately, reporting this delay as part of ack_delay assumes that the implementation timestamps the packet when received at the endpoint. Meaning that if the endpoint actually does the cert verification synchronously, and does not read from the socket until that is complete, then it will need to read kernel receive times. (This also assumes that an endpoint is expected to timestamp the packet on receipt, not on decryption, which is very much how most stacks will do it, but there's not much difference at the moment in the common case between these two times.) I'm tempted to say that we don't try to solve this in the protocol yet. Let's experiment with strategies that we think might work in implementations, share notes, and come back to the spec when we have a good answer. For the time being, we expect that the delays at the clients should not be super common, and even when it is the case, a couple of RTT samples should not significantly sway the RTT. I'm ok with that.
NAME I think we do not need require endpoints doing that. We do not even require endpoints to verify certificate chain asynchronously, meaning that we allow endpoints to simplify their implementation at the risk of performance penalty. I'd argue that not capping ack_delay during the handshake is strictly better than capping the value all the time, assuming that the client's clock is not drifting. That is because the server would have a better RTT estimate when the client reports the correct numbers. If the client fails to do so, it'd be the same as status quo, where RTT estimate would become larger than the actual value. All that said, I am totally fine with leaving this an open area that endpoints can experiment.
Summarizing my thoughts after discussing this with NAME and thinking about this with NAME Apologies for being verbose, but I'm trying to summarize from the top. Problem: At a high level, the problem we are discussing here is that an endpoint buffers packets, because it isn't able to read them or write acks for them, and it might take a while to dequeue the buffer. As a result, the acks can be sent much later than when the packets were received at the endpoint, causing the ackdelay to potentially be much higher than maxackdelay. This inflates the smoothedrtt and rttvariance, inflating the PTO, which can cause handshakes to fail (if a "handshake timeout" period is employed). The problem is that the receiver endpoint can take arbitrarily long to respond. The ackdelay variable is meant to signal precisely this -- we tout it as an important distinction from TCP -- and it is useful here, except that it is limited to maxackdelay in the various RTT estimations. This was done deliberately, allowing for any additional delays to become part of the path delay, with the expectation that such additional delays are outside of the control of the QUIC endpoint and are likely to be recurring. While that remains true, the issue that we are discussing here is an extreme case that occurs during the handshake and is not expected to be recurring. Proposed Solution: We use the ack delay signal during the handshake, and acknowledge that we might need to treat ack delays reported during the handshake period as slightly special. Specifically, we can ignore the maxackdelay, but we still limit the RTT sample to >= minrtt. An endpoint does this until the handshake is confirmed_. That is, until the endpoint is certain that the peer has 1RTT read and write keys and therefore does not have to buffer any more packets for lack of keys. Also: Separately, the recovery spec currently says don't arm PTO on ApplicationData until . This should be . This is because handshake completes at a client when it sends a CFIN, after which the client can send 1-RTT data. The CFIN however might get lost and the server might buffer the 1-RTT data until it receives a retransmission of the CFIN. Changing this to makes it so that no PTO is armed on ApplicationData until there's no Initial or Handshake data left in flight.
Regarding NAME last point, I've changed the moment that the client starts arming PTO timer for ApplicationData PN space from handshake complete to handshake comfirmed. As Jana states, that is the correct thing to do based on the rationale that we have been using to explain why it should not be armed.
FWIW, I was really concerned about that last change. Aside from it making this a non-editorial change, the way that we avoid handshake deadlock depends very much on HANDSHAKEDONE being followed by probes. But as the server confirms and completes the handshake at the same time, I've convinced myself that this is OK. (That this is what our stack did, largely by accident, is vindication of sorts...) I would encourage people to sit down and run the analysis on that change to convince themselves that this is not reintroducing the handshake deadlock. The other part of the proposed solution needs to be slightly different. NAME proposes that we ignore prior to confirming the handshake. I think that's good, but it is not completely correct, and nor would it be sufficient. We need to ignore for acknowledgements of packets that were sent prior to handshake confirmation. Those are the ones that might have been buffered until keys were ready. In order for that to work, we have two new requirements on packet processing for buffered packets: endpoints MUST process and acknowledge (or discard) any packets that they might have buffered during the handshake within of completing the handshake endpoints MUST report an ACK Delay that is based on when a packet is received, not when it was processed
Regarding , I think NAME comment is accurate with possibly one slight modification. We might want to say that ACK Delay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. The observation here is that any unintentional delay (e.g., decryption time, payload processing time) should be considered part of RTT, because that is unavoidable. And the added bonus of defining this way is that then there would be no change required for QUIC stacks that synchronously process the TLS handshake messages.
Just FWIW, I might point out that we are actually making the design streamlined, by improving consistency in how we handle ack delay. The new rule is: The receiver reports the sum of the controlled delays by using the delay field of an ACK frame. When the sender knows what the maximum of that controlled delay is, the sender ... uses that value (i.e. maxackdelay) to arm the PTO timer, and uses that value (i.e. maxackdelay) to calculate the RTT, because if the value of the delay field is greater than the known maximum, the difference is considered as uncontrolled delay and therefore has to be accounted as part of the path delay. When the sender does not know what the maximum of that controlled delay is, the sender ... does not arm the PTO timer, because it does not know when it can expect an ACK to arrive, and calculates the RTT without the capability of correcting misreported controlled delay. It's simply that up until now, we have misused max-ack-delay in the last bullet point case.
This is a good outcome, and I agree with Kazuho. However, I am going to request that someone (not me preferably) take this to the mailing list to confirm with a wider group.
NAME : I agree with what you've spelled out above (). It is close to what I had in mind, but I didn't do a good job of even spelling out fully what I was thinking.
Posting here, FTR. This got no pushback and support, FWIW. Note that this resolution makes this issue a design one, so I'm removing the editorial tag.
NAME noted:
This should be marked design.
I'm trying to fully digest everything above, so I apologize if I missed something important. The idea that you wouldn't acknowledge packets that can be processed while waiting for keys for queued packets is clearly unrealistic for a variety of reasons, so we should definitely remove any text that assumes otherwise. I'm unclear about why moving from handshake complete to confirmed is an improvement, given the example you cite seems to imply the server is retransmitting unnecessary data due to client delays, and for the server complete and confirmed happen at almost identical times. Prior to CFIN or 1-RTT being acknowledged, the client knows the server has only one of the two keys. PTO will retransmit the CFIN on the first PTO and the 1-RTT data on a subsequent PTO(assuming the first PTO doesn't send both). I agree we expect HANDSHAKEDONE to make this irrelevant in practice, but I can't reason about why this change is an improvement based on the problem description. Can someone lay that out for me? I believe we should make normative changes to transport along the lines of S2. Specifically, I think we should say receivers SHOULD include the delay from the time an undecryptable packet is buffered to the time it's acknowledged in the ackdelay, as well as processing all buffered packets for a given encryption level before sending an ACK. Ideally, a receiver would process buffered packets after the full datagram being processed as well, since that allows senders to fully mitigate this issue, but we could leave that to a description, rather than normative text if people prefer. This provides more information to the sender, which it can ignore if it wants, and is much better than sending an Ack Delay of 0us in that case. It's a bit odd that we say a receiver SHOULD set the ack delay field to 0 for Initial and Handshake packets today.
I think it is because the client cannot make an assumption on how long it would take for the server to verify the client's TLS handshake messages being sent using Handshake packets. Important thing to note here is that it could not just be Finished. It might include client certificates. Theoretically, client's handshake transcript might be too large so that it requires multiple round-trips to be sent. Regarding ack_delay, I might not sure if I fully follow. But I think I agree that it would become reasonable for an endpoint to send a Handshake Ack with a non-zero ack-delay, if it buffered the handshake packet that was undecryptable at the moment that packet was received, and later processed it. It is a natural outcome of us changing the definition of ack-delay to the sum of undecryptable buffering delay and acknowledgement delay. As Initial packets would always be decryptable, I think "SHOULD set ack delay to 0" rule can remain for Initial packets.
Specifically, is the concern is that the client will have it's Handshake data acknowledged and will spuriously PTO 1-RTT packets a few times over while waiting for the server to make 1-RTT receive keys available? Is this a problem that's been observed? It seems like it would in practice require client certs, since verifying only a CFIN should be very fast. Making the change to wait for handshake confirmed to PTO 1-RTT data doesn't seem harmful given the existence of HANDSHAKEDONE, but I'm also not sure it's going to matter much in practice. It also adds an exception to the current rule that PTO is armed when the client isn't sure the server has completed address validation or there are bytes in flight. If the Handshake data has been acknowledged, but the 1-RTT data hasn't, you'd have bytes in flight but not arm the PTO if I understand correctly. Given there's still a SHOULD about sending packets from multiple PN spaces on PTO, including Handshake + 1RTT, I think this change will perform well, but I do have apprehension about seemingly small but subtle changes like this late in the process. Agreed, the potential for delay is only for the first Handshake and 1-RTT ACKs. My summary: The issue is that ACKs for Handshake and 1-RTT packets could be delayed because the peer doesn't have the keys(either because it takes time to derive them, or it's missing crypto data to derive the next key). This creates two potential issues(neither of which effects correctness, but could be suboptimal): Packets may be acknowledged later than expected, causing the peer to spuriously PTO. The first RTT sample in Handshake and 1-RTT packet spaces could have substantial delay in excess of the 'no ack delay' in Handshake or maxackdelay for 1-RTT, which inflates SRTT and RTTVar. Changing when to arm PTO for 1-RTT data mitigates one case of the 1st issue(the server being slow to make 1-RTT receive keys available), but I don't believe it helps otherwise with the issue, such as delays in generating Handshake keys. Are there any other design changes(besides complete->confirmed) being considered or suggested for the first issue that I'm missing? If there's no loss, the second issue fixes itself without harm, which is nice. And it's not as bad for ApplicationData, since that will have a non-0 maxackdelay, which may be enough to absorb the endpoint delay generating keys or doing cert verification, though it won't typically be enough to adjust for a packet being lost and the data being PTO'd and then that making keys available. I think Jana's suggestion ackdelay to including the time packets were buffered is sensible, and ignoring maxackdelay for the first sample in a PN space seems relatively safe given at least one prior sample. But what if there are no Initial ACKs received and ACK with the large ack_delay is the first ACK of the connection? I guess you're stuck with one bad sample and hope the next is better? The one change we've found mitigates this issue entirely is coalescing packets from the next PN space(Handshake or 1RTT) and then processing the full coalesced packet before processing all buffered packets and sending an ACK. That ensures there's one new packet being acknowledged and you get a non-delayed RTT sample. It doesn't seem onerous to add a "SHOULD process all packets in the current datagram and all buffered packets before sending an ACK to avoid generating multiple inflated RTT samples" to me?
NAME : I'm yet to process the messages above -- but this issue was covered in .
NAME I think the issue is about principles. I tend to agree that it would be fine for an endpoint to arm PTO timer for 1-RTT packets, when it knows that the peer is going to be able to handle them. However, if we are going to talk about such cases, then I might point out there are other similar cases that are missing in the draft, that might be as important as the case you point out: A server can arm 1-RTT PTO timer when an abbreviated TLS handshake (a.k.a. resumption) is being used. This is because a server to assume that the client would obtain 1-RTT read keys almost immediately. A client can arm PTO timer for 0-RTT packets, assuming that the server would be resuming the connection. My view is that all these are "optimizations" that an endpoint might attempt, based on the knowledge of the form of the TLS handshake being used. But they are just optimizations. I think it would be fine to talk about such optimizations, but at the same time think that we should provide safe and principled rules as baseline, so that endpoints following the recommendations do not end up sending bunch of unnecessary packets. Yeah, I tend to think this as a bug fix. Existing PTO rules seem to assume that the client would spend time in processing the Handshake packets, but the server would not. But the reality is more nuanced, as pointed out by the client-certificate case as well as the examples shown above. And in case where the assumption is too aggressive (the client-cert case), we might see a number of unnecessary packets being sent. All that said, I do think that we should try to minimize the impact of the changes that we introduce to the drafts. That's why I think that something along the lines of "SHOULD NOT arm PTO timer for ApplicationData packets until the handshake is confirmed, unless the sender can assume that the peer would be capable of processing ApplicationData packets when they arrive." Clients arming PTO for 1-RTT packets when client-certificate is not used would fall into this exception, and therefore the design change would not affect the existing QUIC stacks, assuming that they do not use client certificates. Stacks that do use client certificates have to have this fix applied.
NAME That's correct. The resolution proposed by NAME does not mitigate that issue. By including queuing delay of unencryptable packets into ackdelay, the proposed resolution improves the quality of RTT samples. But as you say, it does not mitigate the case where that would be the first sample. I'm not sure if doing that mitigates the problem. The proposed resolution recommends endpoints to send ACK for a handshake message before spending time to process it. As an example, a client would ack the Handshake packets before starting to verify the certificate chain. And after that, it would send a Handshake packet (that contains ClientFinished only, no ack will be included), and a 1-RTT packet acking the 0.5-RTT data sent by the server. If the Handshake ack reached the server, then the server would have a good minrtt estimate. If it did not, then, the next sample that the server would receive is the ack for the 0.5-RTT packet(s). As the servers do not arm the PTO timer for 0.5-RTT data, it is hard to have a protocol mechanism that prevents the first 1-RTT ACKs sent by the client from having a ack delay field that reflects the amount of time spent for verifying the certificate chain.
I created a separate issue() and PR() for the inflated RTT problem, since the solution seems fairly disjoint.
I agree that if you have both sets of keys, the transport should send an ACK and not wait for all async crypto processing to complete. This is similar to receiving a GET request at a proxy. The proxy acknowledges the GET immediately even if it has to go to a backend to respond. But that doesn't prevent greatly inflated RTT samples when the keys aren't immediately available.
It's also above, but the inflight PR only attempts to address part of the problem, not the inflated RTT issue that I mentioned halfway through the issue. Also, I'm finding it confusing to track what problems are trying to be solved and what ones are already largely solved, so I was hoping restating my understanding would allow you to correct me if I'm misunderstanding something.
I think we should flag this as design and run the process. It's hard to pin down exactly what changes here substantively, but it's worth getting the eyes. I'd like to bring this slightly tricky issue under the wg's eyes. Discussions so far have yielded a strategy that I propose further below. This summarizes conversations in 3821 and 3874 (and the outcomes might touch 3889 ). The purpose of this email is to see if we can agree on the strategies outlined here, so we can move forward on the PR there and/or other associated issues and PRs. Problems: During the handshake, an endpoint can receive packets (carrying data or acknowledgements) that it cannot decrypt because the requisite keys are not yet available. This can happen because the crypto machinery at the endpoint is waiting for earlier packets to be received to yield the keys. Similarly, a client might not be able to read 1-RTT packets received from the server because it is waiting for certificate validation to complete. The assumption in the TLS specification is that when an endpoint is busy doing something (certificate validation or key generation for instance), it won't be able to respond to received packets with acknowledgements (Section ). Contrary to the assumption in the spec, Chrome and Firefox both process the certificate chain asynchronously, and therefore acks handshake packets immediately, even the ones that arrive after they have started verifying the certificate chain. But they buffer 1-RTT packets, and therefore the issue exists for 1-RTT packets. As a result, there's potential delay, sometimes significant, in responding to some packets during the handshake. There are two side effects of this. S1. First, this means that an endpoint that runs a PTO timer and retransmits data for 0-RTT and 1-RTT packets while the handshake is still in progress might be doing this quite unnecessarily. The peer may well have received this data and have queued it, waiting to be able to decrypt it. S2. Second, acks for these packets that are buffered and then later decrypted and read can be sent much later than when the packets were received at the peer, causing the ackdelay to potentially be much higher than the advertised maxackdelay. The endpoint ignores ackdelay larger than maxackdelay, and this inflates the smoothedrtt and rttvariance, potentially inflating the PTO significantly (which can cause handshakes to fail if the endpoint employs a "handshake timeout" period). Proposed Solutions: S1. To solve the first problem, we already have a condition in the recovery draft to not arm the PTO on 1-RTT data until the handshake is complete (Section ). This is not quite adequate, since it still allows a client to arm a PTO timer on 1-RTT data on sending a CFIN and 1-RTT data, before ensuring that the server has received the CFIN. The proposed solution here is to change this condition from "handshake complete" to "handshake confirmed". Meaning that an endpoint only arms a PTO timer on 0-RTT or 1-RTT data when it knows that the peer has the keys necessary to read and write in 1-RTT, and can therefore respond with an ACK. We think this holds up, but we'd like to get this under everyone's eyes, to ensure that the discussion isn't missing subtle deadlock issues. S2. To solve the second problem, the proposal is to use ackdelay as reported by the endpoint that buffers the data appropriately. ackdelay is the amount of the time the information (regardless of that being the packet itself or the acknowledgement) being buffered intentionally. For this, we need the following: endpoints MUST report an ackdelay that is the sum of the controlled delay (i.e. the duration of the data and the ack being buffered) endpoints ignore maxack_delay for acknowledgements of packets that were sent prior to handshake confirmation. Those are the ones that might have been buffered at the peer until keys were ready. In addition, we explicitly recommend endpoints that asynchronously process handshake messages to buffer 1-RTT packets, instead of dropping them, as dropping packets has significant impact on congestion control. Note that this matches what those implementations already do. Thoughts? jana

Discussion:
As keys at a given encryption level become available to TLS, TLS indicates to QUIC that reading or writing keys at that encryption <del> level are available.  These events are not asynchronous; they always occur immediately after TLS is provided with new handshake bytes, or after TLS produces handshake bytes. </del> <ins> level are available.  While generating these keys, an endpoint SHOULD buffer received packets marked as protected by the keys being generated, and process them once those keys become available.  If the keys are generated asynchronously, an endpoint MAY continue responding to the received packets that were processable while waiting for TLS to provide these keys. </ins> TLS provides QUIC with three items as a new encryption level becomes available:

5. As with TLS over TCP, QUIC protects packets with keys derived from <del> the TLS handshake, using the AEAD algorithm negotiated by TLS. </del> <ins> the TLS handshake, using the AEAD algorithm AEAD negotiated by TLS. QUIC packets have varying protections depending on their type: Version Negotiation packets have no cryptographic protection. Retry packets use AEAD_AES_128_GCM to provide protection against accidental modification or insertion by off-path adversaries; see retry-integrity. Initial packets use AEAD_AES_128_GCM with keys derived from the Destination Connection ID field of the first Initial packet sent by the client; see initial-secrets. All other packets have strong cryptographic protections for confidentiality and integrity, using keys and algorithms negotiated by TLS. This section describes how packet protection is applied to Handshake packets, 0-RTT packets, and 1-RTT packets.  The same packet protection process is applied to Initial packets.  However, as it is trivial to determine the keys used for Initial packets, these packets are not considered to have confidentiality or integrity protection. Retry packets use a fixed key and so similarly lack confidentiality and integrity protection. </ins> 5.1.

separation between QUIC and TLS; see key-diversity. The KDF used for initial secrets is always the HKDF-Expand-Label <del> function from TLS 1.3 (see initial-secrets). </del> <ins> function from TLS 1.3; see initial-secrets. </ins> 5.2. <del> Initial packets are protected with a secret derived from the Destination Connection ID field from the client's Initial packet. Specifically: </del> <ins> Initial packets apply the packet protection process, but use a secret derived from the Destination Connection ID field from the client's first Initial packet. </ins> <del> The hash function for HKDF when deriving initial secrets and keys is SHA-256 SHA. </del> <ins> This secret is determined by using HKDF-Extract (see Section 2.2 of HKDF) with a salt of 0xafbfec289993d24c9e9786f19c6111e04390a899 and a IKM of the Destination Connection ID field.  This produces an intermediate pseudorandom key (PRK) that is used to derive two separate secrets for sending and receiving. The secret used by clients to construct Initial packets uses the PRK and the label "client in" as input to the HKDF-Expand-Label function to produce a 32 byte secret; packets constructed by the server use the same process with the label "server in".  The hash function for HKDF when deriving initial secrets and keys is SHA-256 SHA. This process in pseudocode is: </ins> The connection ID used with HKDF-Expand-Label is the Destination Connection ID in the Initial packet sent by the client.  This will be

packet after receiving a Retry packet, where the Destination Connection ID is selected by the server. <del> The value of initial_salt is a 20-byte sequence shown in the figure in hexadecimal notation.  Future versions of QUIC SHOULD generate a new salt value, thus ensuring that the keys are different for each version of QUIC.  This prevents a middlebox that only recognizes one version of QUIC from seeing or modifying the contents of packets from future versions. </del> <ins> Future versions of QUIC SHOULD generate a new salt value, thus ensuring that the keys are different for each version of QUIC.  This prevents a middlebox that recognizes only one version of QUIC from seeing or modifying the contents of packets from future versions. </ins> The HKDF-Expand-Label function defined in TLS 1.3 MUST be used for Initial packets even where the TLS versions offered do not include TLS 1.3. <del> The secrets used for protecting Initial packets change when a server sends a Retry packet to use the connection ID value selected by the server.  The secrets do not change when a client changes the </del> <ins> The secrets used for constructing Initial packets change when a server sends a Retry packet to use the connection ID value selected by the server.  The secrets do not change when a client changes the </ins> Destination Connection ID it uses in response to an Initial packet from the server.

packet; the client has to rely on the exchange that included the Retry packet for that property. <del> test-vectors contains test vectors for packet encryption. </del> <ins> test-vectors contains sample Initial packets. </ins> 5.3.

using the TLS_AES_128_GCM_SHA256 cipher suite, the AEAD_AES_128_GCM function is used. <del> Packets are protected prior to applying header protection (header- protect).  The unprotected packet header is part of the associated data (A).  When removing packet protection, an endpoint first removes the header protection. All QUIC packets other than Version Negotiation are protected with an AEAD algorithm (AEAD).  Prior to establishing a shared secret, packets are protected with AEAD_AES_128_GCM.  Retry packets use the AEAD for integrity protection only, as described in retry-integrity. Initial packets use a key derived from the Destination Connection ID in the client's first Initial packet; see initial-secrets.  This provides protection against off-path attackers and robustness against QUIC-version-unaware middleboxes, but not against on-path attackers. </del> QUIC can use any of the cipher suites defined in TLS13 with the exception of TLS_AES_128_CCM_8_SHA256.  A cipher suite MUST NOT be negotiated unless a header protection scheme is defined for the

deploy a new cipher suite.  This also applies to TLS_AES_128_CCM_8_SHA256. <ins> When constructing packets, the AEAD function is applied prior to applying header protection; see header-protect.  The unprotected packet header is part of the associated data (A).  When processing packets, an endpoint first removes the header protection. </ins> The key and IV for the packet are computed as described in protection-keys.  The nonce, N, is formed by combining the packet protection IV with the packet number.  The 62 bits of the