Change:
I've been debugging a connection failure that seems to be caused by a key update. Here's what's happening: At some point during the connection, an endpoint decides to update its keys. After the key update, it doesn't send any (ack-eliciting) packets. If it sent non-ack-eliciting packets during this time, these packets are lost. After a while (e.g. 3 PTOs) it drops the old keys. The peer never received a packet in the new key phase, so it continues sending using the old key phase. Because the keys are dropped, these packets will all end up being undecryptable. Note that this is not an uncommon situation: An endpoint downloading a large object might initiate the key update, but not have anything to send but acknowledgements. says that receive keys MAY be discarded after 1 PTO: ... and SHOULD be discarded after 3 PTO: It seems like this advice is not safe. The deadlock could be avoided if old keys are kept until an ACK has been received for a packet encrypted with the new keys.
Just wondering, but isn't section 6.5 about receiving key updates? I might argue that the the moment of "key update" referred to by that section is the moment the endpoint receives a packet using a new key phase. If we interpret that way, then this is a non-issue, I think? FWIW, quicly does something along these lines.
I read this text as being about receiving packets during a key update, no matter who initiated the update. Even if you interpret as only applying when the peer initiated the update, that still leaves us lacking any text when it’s safe to discard old keys for the endpoint that initiated the update. As the deadlock shows, there’s no duration that can be considered safe.
Right. As stated, my argument is that the receive key of generation N is not supposed to de dropped until the endpoint receives a packet protected using a receive key of generation N+1. Once a packet protected using a receive key of generation N+1 is received, the endpoint is recommended to drop key of generation N within the 3 PTO window. That is what the text quoted above says: An endpoint SHOULD retain old read keys for no more than three times the PTO. After this period, old read keys and their corresponding secrets SHOULD be discarded.
I'm a bit uncomfortable tying the trigger to the peer's own key update, even though they're required to update in step. Would it make more sense to say you don't discard until you've received an acknowledgement for a packet sent with the new keys? That already has to be tracked for the maximum frequency of key updates, so it would make sense to hang this mechanism on it as well. If you know the peer has received a packet from you in the new phase, they'll be updating soon enough, but you know they have read keys which is the real issue.This seems like what we intended to me.

Discussion:
it uses for receiving packets.  These keys will be needed to process packets the peer sends after updating. <del> An endpoint SHOULD retain old keys so that packets sent by its peer prior to receiving the key update can be processed.  Discarding old keys too early can cause delayed packets to be discarded.  Discarding packets will be interpreted as packet loss by the peer and could adversely affect performance. </del> <ins> An endpoint MUST retain old keys until it has successfully unprotected a packet sent using the new keys.  An endpoint SHOULD retain old keys for some time after unprotecting a packet sent using the new keys.  Discarding old keys too early can cause delayed packets to be discarded.  Discarding packets will be interpreted as packet loss by the peer and could adversely affect performance. </ins> 6.2.

Phase bit alone can be used to select keys. An endpoint MAY allow a period of approximately the Probe Timeout <del> (PTO; see QUIC-RECOVERY) after a key update before it creates the next set of packet protection keys.  These updated keys MAY replace the previous keys at that time.  With the caveat that PTO is a subjective measure - that is, a peer could have a different view of the RTT - this time is expected to be long enough that any reordered packets would be declared lost by a peer even if they were acknowledged and short enough to allow for subsequent key updates. </del> <ins> (PTO; see QUIC-RECOVERY) after receiving a packet that uses the new key generation before it creates the next set of packet protection keys.  These updated keys MAY replace the previous keys at that time. With the caveat that PTO is a subjective measure - that is, a peer could have a different view of the RTT - this time is expected to be long enough that any reordered packets would be declared lost by a peer even if they were acknowledged and short enough to allow for subsequent key updates. </ins> Endpoints need to allow for the possibility that a peer might not be able to decrypt packets that initiate a key update during the period

sufficient time could lead to packets being discarded. An endpoint SHOULD retain old read keys for no more than three times <del> the PTO.  After this period, old read keys and their corresponding secrets SHOULD be discarded. </del> <ins> the PTO after having received a packet protected using the new keys. After this period, old read keys and their corresponding secrets SHOULD be discarded. </ins> 6.6.