Code change:
This is attempt to write down NAME 's proposal to This was surprisingly hard to write, and undoubtedly could be improved. There are cleaner ways to solve this problem, but this one is lowest-impact to the existing spec.
NAME thanks for the new text, that helped. I think we've resolved outstanding issues, unless my new text has created new problems. 2 possible ways to enlarge this solution: should we describe this attack in Security Considerations? As these MAYs are an effort to counter something in particular, perhaps we should discuss. it would clean up some accounting issues if each NCID frame MUST set Retire Prior To to more than the maximum continuous sequence number that it had recieved an RCID for. Worthwhile?
That's interesting to send at first glance, but then poses the question of what the recipient of NCID should do when it sees an RPT value change. There's a difference in behavior between "I've seen these" and "you need to send these."
I think this PR is sufficient to both document the issue and provide some minimum guidance, so I'd suggest we editorialize a bit and then move forward with a solution along these lines.
It seems all the discussion of the problem is happening here and not on the issue, so I will propose my alternative solution here: I know folks have been resistant to changing state based off receiving an acknowledgement of a packet/frame they sent, but what if we changed how the RetirePriorTo works, and just required the packet containing the frame to be acknowledged, instead of requiring the RCID to be sent? Would this solve the problems? This would mean the receiver of the NCID shouldn't have to track any additional state beyond what it takes to acknowledge the packet (which they have to do anyways); it just immediately throws away the state. And the RCID packet would only be used when the endpoint that was given the CID chooses to retire it.
NAME I think this PR has been an attempt to clarify the "leaky" approach, that tries to clarify the amount of state that an endpoint should retain. There are indeed discussion of what that amount is, but I think it is essential to have such detailed discussion on the PR in order to determine exactly what a particular approach would be like. To paraphrase, I think we are making process on this PR to describe a specific approach. If we are to consider something different, I think that discussion should happen on the issue (and emerge as a separate PR).
Ok, trying to put this one in the ground. I think the only remaining dispute is between NAME and NAME on whether activeconnectionid_limit is a sufficient number of RCIDs to remember.
Reading through this proposal, I fear that this would be very difficult to implement. If the ACK for a RETIRECONNECTIONID frame is lost, and endpoint cannot safely send a NEWCONNECTIONID frame with a RetirePriorTo, since this would cause the peer to have more than RETIRECONNECTION_ID frames in flight. In order to safely use RetirePriorTo, this endpoint then would have to track receipt of the ACK frame. I'd very much like to avoid the complexity associated with that.
I finally caught up with this issue, and I think that implementer advice is my preferred way forward. To that end, I like this PR as the resolution.
OK -- there has been an enormous amount of discussion. The current state of the PR is that all RCIDs will eventually be sent, but if there are more than activeconnectionid_limit, some may be delayed by an RTT or more. If it gets out of hand the RCID sender can throw a connection error. I believe this reflects what NAME articulated a few dozen messages ago, and in the thread above I have pseudocode for a lightweight implementation of this (i.e. essentially no additional resources for an arbitrarily large backlog of RCIDs). The advantage of this approach is that it is lightweight and is very subtle change to behavior that will not will not affect interoperability at all or break the existing contract of a Retire Prior To. There are two alternatives that have some traction: 1) Just bite the bullet and do . This breaks interoperability but is a clean architectural change that addresses the root of the problem. I would be very comfortable with this, and prefer it to other things that break the existing semantics. 2) NAME "leaky" approach that allows endpoints to simply not send some RCIDs at all if there are too many pending. I used to prefer this, but given my existence proof above I'm not as excited about it anymore. It breaks the contract that is in draft-27 instead of merely bending it. I hope that covers the decision space. I would like to put this issue out of its misery.
I'm reluctant to blow away the whole PR with brand new text, so in this comment I will propose four entirely new paragraphs that IMO says what needs to be said. Either the current PR, Martin's revision, or this proposal could serve as the basis of the eventual change: If a peer sends large numbers of NEWCONNECTIONID frames that increase Retire Prior To, and/or acks of packets that contain RETIRECONNECTIONID are lost, the state required at the RETIRECONNECTIONID sender can grow without regard to its activeconnectionidlimit. Therefore, endpoints SHOULD take steps to bound the state associated with needed RETIRECONNECTIONID frames while ensuring that it eventually transmits all required RETIRECONNECTIONID frames. For example, it might limit voluntarily retirement of sequence numbers if it has not received enough acknowledgments of packets containing previous retirements. It might also restrict the RETIRECONNECTIONID frames in flight to a single packet in order to simplify tracking of what is in flight, what needs retransmission, and what has been acknowledged. An endpoint MAY treat having too many connection IDs to retire as a connection error of type CONNECTIONIDLIMITERROR. The threshold for this error SHOULD be at least twice the endpoint's advertised activeconnectionidlimit. Endpoints SHOULD NOT issue updates of the Retire Prior To field prior to receiving all of the RETIRECONNECTION_ID frames for the previous update to Retire Prior To. ** If people much prefer this or MT's revision, I'll update the PR accordingly.
I took NAME 's suggested text. I would prefer a little advice on how to implement the lag, but I can live with this text.
LGTM.
When an endpoint sends a RETIRECONNECTIONID frame carrying a connection ID that has been active until that point, that endpoint is expected to open room for accepting a new connection ID. That means that if the peer does not acknowledge the packets that contains RETIRECONNECTIONID frames, but keeps on sending NEWCONNECTIONID frames with increasing values in the Retire Prior To field, the number of unacknowledged RETIRECONNECTIONID frames (or to be precise, the number of unacked retiring connection IDs) keeps on increasing.
To address the issue, I think we need to do one of the following: State that the attack exists, and that QUIC stacks should implement mitigations. Introduce MAXCONNECTIONIDS frame. The reason we do not have similar issue in stream concurrency control is because the credit is controlled by MAX_STREAMS frames, rather than using FIN (or reset) as an implicit signal to indicate availability of new credit. We could do the same for connection IDs.
Mathematically, this might be unbounded, but it's more likely that in practice the limit is as it would be unlikely that unacknowledged RETIRECONNECTIONID frames would remain so once more NEWCONNECTIONID frames start arriving. More so as the issuer of those NEWCONNECTIONID frames has to have received the RETIRECONNECTION_ID frames. Adding new frames for this seems a little unwieldy, so I'd prefer to just acknowledge that the problem exists. Note also that Retire Prior To has the effect of making this limit squishy in other ways. (This is a design issue.)
I'm not sure I understand the attack here. Why would an implementation need to keep track of acknowledgements of RETIRECONNECTIONID frames (other than for purposes of loss recovery, but then this applies to any retransmittable frame type)? An endpoint sends a RETIRECONNECTIONID frame when it decides to not use the respective CID any longer. At this point, it can forget about the connection ID altogether. It might make sense to hold on to the stateless reset token for a while (3 PTO?) longer, in order to be able to detect stateless resets, but this is an optimization and wouldn't lead to unbounded state anyway.
NAME I think is fine, assuming that the endpoint would just stop retransmitting oldest RETIRECONNECTIONID frames. That limit would be too stern to be used for detecting potential attacks (and therefore closing the connection with PROTOCOLVIOLATION).
NAME This is about keeping state for retransmission for recovering loss. As stated in , the issue is unique to RETIRECONNECTIONID frames, because this frame type is (ab)used for providing the peer additional credit, compared to other flow control that uses dedicated frame types (e.g., MAX_*) for communicating credit.
Thank you for the explanation, NAME I think I now understood the problem. This would require endpoints to make assumptions about how the peer handles the and how it issues new CIDs. This can get complicated really quickly and leave the connection in an unrecoverable state (or at the least a state with less CIDs than it should have). Introducing a new frame, as NAME suggested, seems a lot cleaner to me.
Just to be clear, while I used RPT as a way of explaining the design issue, I think it is not a requirement. A malicious client can repeatedly migrate to a new address, initiating the use of new CID pairs, at the same time intentionally not acknowledging packets containing RETIRECONNECTIONID frames sent from the server. If the client does that, the number of CIDs that the server has to track for retirement increases as time goes.
Clarification question: If the person sending RCID isn't getting ACKs back, I think we still need to assume that those CIDs are being replaced? If not, then the non-RPT side of this would just mean the person sending RCID frames runs out of CIDs to use and has to stop. So the problem statement is that: this person is running over their maxactivecid limit from the perspective of the person withholding ACKs, but each side has their own view and so the person retiring them stopped counting them as active once the frame was sent (but not acknowledged). And that can happen via RPT as well, just because it means that they're going to be instructed to send the RCID, but the problem is the same either way? (Just want to make sure I'm understanding the problem correctly)
NAME Yes, I think your problem statement is correct. Or to be even more concise, the problem is that the issuer of CIDs is given additional credit when it receives RCID, which happens before the consumer of CIDs drops the state it needs to retain for retransmitting RCID. For the attack to work, an attacker has to let the peer consume and retire CIDs. RPT and intentional migration are examples that makes that happen.
So the issue here is fundamentally that you can induce the peer to send retransmittable frames which aren't subject to some kind of flow control, and this is the only instance we have of that. STOPSENDING and failing to acknowledge the RESETSTREAM appears to have a similar profile, except that the number of RESETSTREAMs you could have outstanding is bounded by the number of streams the peer has opened, which they control. You could execute the same attack at a larger scale by choosing particular regions of stream data to never acknowledge, forcing the peer to keep them in memory for retransmission, but the peer could stop sending until that region gets through (applying backpressure) or simply reset the stream. Since a RESETSTREAM is smaller than the data region and only one is needed per stream, this collapses into the previous case. What's different here is that the endpoint being attacked doesn't choose whether to retire these CIDs and has no mechanism to apply backpressure. It seems like there are a couple paths to resolving this. As you suggest, we could define an explicit frame that grants the additional CID space, and implementations would not send / could withhold this frame until the RCID is ACK'd. This creates a backpressure mechanism and has the side benefit of allowing an implementation to vary the number of CIDs it wants to track over time, but the disadvantage of requiring an extra round trip before CIDs can be replaced in the normal case. We could mimic the "collapsing" behavior above by defining a frame that retires all CIDs up to a given sequence number. Then the implementation under memory pressure chooses to retire a few extra CIDs and collapses the whole set waiting to be retransmitted to a single frame.
NAME I think your summarization of the problem is accurate. Regarding an attacker selectively acking packets carrying STREAM frames, yes, the attack exists, but it the sender can mitigate that problem in certain ways. We already state that in . This issue is the only case that a protocol cannot be implemented properly, with a bounded amount of state. Regarding how we should fix the problem, if we are going make a protocol change, my preference goes to applying the same design principle that we have in other places (i.e. introduce MAXCONNECTIONSIDS frame). I do not like designing a mechanism specific for this purpose, as there is a chance that we might make mistakes. I can also live with what NAME proposed in URL (i.e. encourage leaky behavior).
Similar to NAME I'd prefer to acknowledge the problem and indicate that peers may want to limit the number of unacknowledged RETIRECONNECTIONID frames to the number of connection IDs they support. The other option I can imagine is to make the RETIRECONNECTIONID frame cumulative, which provides a strong incentive to use connection IDs in order. Though we may have to make an exception for the connection ID in the received packet if we did that.
So the bounds on this aren't that terrible. Say you had to retire 1000 connection IDs all at once. You could track each one individually as you send RETIRECONNECTIONID. That's 1000 things to track, which might be more state than you want (it's not 1000 bits because you need to maintain relationships with packet numbers). Or you could just keep strictly limited, no matter how many connection IDs you need to retire. Tracking that, plus what you have been requested to retire should bound the state required to implement retirement. In other words, you do control how much state you commit to retirement, just like everything else.
NAME That's my preferred solution, but in my reading of the spec this is not allowed. We MUST send RCID for each sequence number for which the peer has sent retire-prior-to. I think the language in Section 5.1.2 needs to change.
NAME idea of allowing RCID to be cumulative is also good and computationally cleaner, although we'd need a different frame type or an additional field in the frame (I don't think we can get rid of retiring non-continuous sequence numbers)
I'm happy to file a PR for the NAME or NAME approach based on the overall sentiment.
One purely implementation option: It's always legitimate to repeat a frame. Therefore, if you track only the smallest and largest un-acked RCID sequence numbers, you can just keep retiring the whole range every time you retransmit. ACKs will shrink the range (hopefully to zero). There's no harm beyond unnecessary bytes in retransmitting something a few extra times to avoid excessive state tracking.
NAME While I won't say that's impossible, I'd argue that the design is going to be complex. When CIDs are used in a legitimate manner, they are not going to be retired one by one. A CID is retired when a path using that CID is abandoned. Consider the case where an endpoint has 4 active CIDs (CID0 to CID3), and two paths. One path could be using CID0, the other path could be using CID3. When the endpoint abandon the path that is using CID3, it has to retire CID3 but it cannot retire others. NAME The amount of CIDs that can be sent on wire at a time is restricted by the CWND, therefore there is no guarantee that you can send RCID frames for all CIDs that you want to retire at once. In other words, I think the approach at best collapses into what NAME has proposed (i.e. leaky behavior).
At this point, I also think NAME solution is the best approach, since it is a very minor change, and changing RETIRECONNECTIONID to be cumulative is a larger change than I want to accept at this point. NAME thanks for your examples.
One idea: Can we simplify everything by not requiring sending RETIRECONNECTIONID for CIDs less than the Retire Prior To? RETIRECONNECTIONID would then only be used when a CID was used and needed to be retired so more CIDs could be issued to the user of the CID.
NAME see my comment in the PR. More disruptive to current implementations, and eliminates a useful signal. The current proposal will not matter for well-behaved clients but will protect servers from some attacks.
We haven't implemented this yet, so it's not disruptive to our implementation FWIW I wonder how useful the signal is. Retire Prior To is really saying "These are going away sometime soon, whether you like it or not." to the peer. If an implementation never sent RETIRECONNECTIONID, would that create a problem of some sort, because I suspect someone will do just that.
NAME As stated in URL, this attack can be mounted without using Retire Prior To.
NAME in this case isn't the client limited by activeconnectionid_limit? As a server, if the client has in no way indicated it has accepted my retirement, I'm not going to accept new CIDs. The NCID/RPT case is different because the spec specifically allows the client to NCID speculatively assuming the server will immediately retire the CID.
NAME The attack here is that when a malicious client retires a CID to which the server has responded, that server would retire the CID that it has used on that path, and also provide a new CID to the client. But the client never ACKs the RCID frames that the server sends. As an example, consider the case where a client uses sCID1 on a new path, the server responds on that path using cCID1, the client abandons that path and sends RCID (seq=sCID1). When receiving this RCID frame, the server would send RCID (seq=cCID1), and also send NCID(sCID2). The client intentionally does not ack the packet carrying these frames, but uses sCID2 on a new path, that carries NCID(cCID2), repeating this procedure.
Ah, I see now that my defense doesn't work, because if I reject the NCID I have to kill the connection, and it may just be because of unlucky ack losses. This whole section is not very precisely written and suffers from two-generals problems about when a CID is actually retired. I will revise to account for this, but I'm not sure how.
NAME I agree with NAME that there's no valid thread model without Retire Prior To. If the peer doesn't acknowledge your retirement, then you shouldn't be issuing more CIDs, because you might exceed their limit.
In my opinion, running out of CIDs is not a big crisis. You can always create a new connection with 0-RTT with a relatively small overhead. We should be biased towards simplicity rather than never running out of CIDs.
NAME unfortunately we are cornered by the spec. Consider two well behaving endpoints. I'm fully stocked with CIDs and then retire one with a frame. The peer gets the RCID and issues an NCID, but the ack of RCID is lost. If I don't count the CID as retired, then I MUST close the connection. That's bad, so I have to treat the CID as retired if I sent RCID. If you drop acks at scale, this becomes an attack. I'm really coming around to your cumulative retire idea, but will have another go at this tomorrow.
NAME I think the framing is incorrect. The peer can enforce an endpoint to retire a CID (e.g., by changing RPT or discarding a path). As NAME points out, an endpoint is encouraged issue a new CID at this point, rather than withholding than until the ACK of RCID is received. Note also that RCID that an endpoint sends carries the CID that the peer has issued, while NCID that the endpoint carries the CID that that endpoint issues. Therefore, I am not sure what the rationale for withholding the emission of NCID until receiving an ACK for RCID would be. I agree. An endpoint is not expected to consume CID at high rate. We already discourage such behavior; see URL That's why I'm fine with the leaky behavior that NAME has suggested. NAME As previously stated, I am hesitant to have a new machinery that is unique to this purpose. There is a risk of creating something buggy, both in terms of specification and implementation. My preference goes to accepting the risk (and recommending mitigations, such as the leaky behavior), or reusing a design pattern that we already have (i.e. introduce MAXCONNECTIONIDS frame).
The cumulative idea is very appealing to me, minus a future with multipath. If two or more paths are active at once, I don't know how the peer would know that if the frame was cumulative. The only option would be to ensure the two(or more) CIDs in use are adjacent, which is possible, but may require switching CIDs more than should be necessary.
NAME I would suggest a cumulative RCID as a supplement to the individual RCID. A cumulative one would cleanly handle a lot of cases where there is an unboundedly large list of sequence numbers to retire. If there is a low sequence number that is still in use, the sender can choose to retire it to enable use of the cumulative frame.
I know folks have been resistant to changing state based off receiving an acknowledgement of a packet/frame they sent, but what if we changed how the RetirePriorTo works, and just required the packet containing the frame to be acknowledged, instead of requiring the RCID to be sent? Would this solve the problems? This would mean the receiver of the NCID shouldn't have to track any additional state beyond what it takes to acknowledge the packet (which they have to do anyways); it just immediately throws away the state. And the RCID packet would only be used when the endpoint that was given the CID chooses to retire it.
FWIW, I have filed that fixes this issue without introducing a leaky behavior like . IMO, the observation is that the endpoints can check that the issuer of CIDs is not intentionally introducing gaps in the sequence numbers they generate. We already prohibit senders from introducing such gaps, detecting them on the receiver side would be a good thing to do. And if endpoints start doing that, using a cumulative RCID frame is a sensible solution, as the the cumulative list of retired CIDs at any time can be represented using a SACK-like structure (i.e., max-retired + small list of active CIDs below that max.).
I know that you have characterized this as "leaky", but I'm not sure that I understand exactly what is being leaked. Nothing is lost in , except that the endpoint requesting retirement has to tolerate a potentially extended period without old connection IDs being retired. In case this helps, I have no big preference between and . There is a certain elegance to that might tip me toward that, but it is slightly more disruptive. I prefer either of those to the more disruptive change in , which also has the downside of strongly encouraging connection ID changes in order to preserve a clean separation between active and retired. doesn't force you to drop old connection IDs. However, the encoding of the frame is just too complicated for me to accept. I would prefer forcing a clean separation than having a complex encoding like this. is a non-starter for me for the reasons stated.
NAME In the approach we take in , there is a possibility that the issuer of the CID might not receive the sequence numbers of all the CIDs that have been retired by the peer, when the issuer is issuing new CIDs at high rate and the receiver is also consuming them at a hight rate. That is the consequence of endpoints limiting the amount of inflight RCID frames it tracks. I think we can provide advice so that the "leak" would not be an issue in practice, and assuming that would happen, I am happy with being the solution. I think is cleaner, but I acknowledge that it is more disruptive than . Part of my intent behind creating is to see how a clean fix (in terms of state machinery) would look like, so that it can be compared against . As stated in URL, by itself only partly fixes the problem. I am not against adopting along with though. is also a partial fix, I also think that it is a non-starter.
I think that I understand your point; thanks for your patience. I didn't really classify it as a leak though, so the choice of words threw me a little, it's more of a lag. Let's say that you have a pathological endpoint that sends an infinite series of NEWCONNECTIONID frames with Sequence Number N and Retire Prior To of (N-1). That's legal and will never violate any limits. But unless they maintain acknowledgments for RETIRECONNECTIONID frames at a similar rate (which, to be fair should be easy as for the same values), they could get far ahead of their peer. I don't think that it is strictly a leak, just a lag between the connection IDs being pushed and the connection IDs being successfully retired. Without or something else this could be problematic. And I now understand why is really just an orthogonal refinement, though it might make other defenses less likely to be required by making zero in cases where an endpoint isn't using discretionary retirement. The defense in is to make effectively constant, no matter how many frames need to be retired. My main objection is really that it is a maximal design where only a minimal one is warranted. It's probably worth pointing out that there is also the simplest defense: If you can't keep up with RETIRECONNECTIONID, stop talking to the peer that does this to you. This is just another example from that category of abuse we don't have specific protocol mechanisms to mitigate.
I agree that I'd like to see a minimal solution, so I'm leery of heading towards . In order for a peer to need to send a lot of RETIRECONNECTIONID frames without Retire Prior To, a lot of 5-tuple + CID changes need to be coming in. One way to rate limit that naturally is to stop giving out CIDs if the peer hasn't acknowledged your RETIRECONNECTIONID frames. ie: Limit the number of NEWCONNECTIONID frames in flight to: So if the peer stops acknowledging your retirements, you stop giving them new CIDs. I think this avoids the need for any other limits? I added 1 in my example so one NCID could be sent even if all of the peer's CIDs had just been retired, though that concern disappears if we decided to also do something like
What a mess we've made! Is there interest in doing a Zoom about this? Maybe 0900 Tokyo/1100 Sydney/1700 Seattle/2000 Boston on Tue/Wed? The root cause is that the flow control mechanism isn't all that well designed. IMO is the actual correct fix to this problem, but as people say, it's disruptive. uses ACKs as an implicit signal, which seems like a problem. has a different implicit signal, and I don't think it solves the case where the client is withholding acks for unsolicited RCIDs. I really think we should just use or depending on our appetite for changing the wire image. is great for the RCID sender, and if the receiver won't miss any RCIDs unless it's doing pathological stuff or the ack loss patterns are tremendously unlucky. , OTOH, is a comprehensive fix to the problem.
I think with some added text about limiting outstanding NCIDs when there are lots of RCIDs would be sufficiently robust.
I'm fine with trying to see if we can reach an agreement on .
I'd prefer over . I can't really see why it would more disruptive than . The problem with is that it will likely work find in most cases, but there are some corner case (when the ACKs for RETIRECONNECTIONID frames are lost), in which things will break. Properly accounting for these corner cases requires some non-trivial tracking logic in order to avoid risking the connection being closed with a CONNECTIONIDLIMIT_ERROR. On the other hand, seems to be conceptually simpler. I'd rather implement a few changes to my frame parser than introduce some ACK-tracking logic.
Just to give another example of how an endpoint might end up required to track the retirement of more than activeconnectionid_limit, when talking to a well-behaving peer: A server tries to supply three active CIDs to the peer. At the moment, it has provided CID0, CID1, CID2 to the client. The client sends RCID(CID1), RCID(CID2). The server provides NCID(CID3), NCID(CID4). The two NCID frames are delivered to the client, but the ACK for the RCID frames gets lost. The client decides to probe two paths concurrently, by using CID3 and CID4, but shortly after the packets are sent using those paths, the underlying network for those two paths disappear. The client now needs to make sure that RCID frames carrying 4 CIDs reach the peer. In this example, we might argue that it would be possible for the client to figure out that CID3 and CID4 were provided as substitutions for CID1 and CID2. But do we want to implement logic that detects such condition into our code? Note that tracking of substitutions can become tricky; if RCID(CID1) and RCID(CID2) were sent in different packets, it would be impossible for the client to see if CID4 was provided as a supplement for CID1 or for CID2.
I agree with NAME that if we go with the limit should be higher than .
I finally caught up with this issue, and implementer advice is definitely my preferred way forward. We're going to need some complexity to (and some magic numbers) no matter what, let's do this while limiting mechanism to a minimum. To that end, I agree with as a good resolution.
While increasing the limit will certainly help, this introduces a really unpleasant flakiness in the protocol. QUIC, so far, has the property that no amount or pattern of packet loss will lead to a protocol violation (it might lead to a connection timeout if all packets are blackholed, but that's a different kind of error). This means that when running QUIC in production, screening for (transport-level) errors will be very useful for finding bugs in implementations. My fear regarding is that no matter how high we choose the limit, there will always be a loss pattern that will trigger a protocol violation.
NAME Thank you for raising the concern, I think the recommended behavior when exceeding that limit is to stop sending RCID frames for some sequence numbers rather than triggering a protocol violation. That's actually what I had in mind hence called it "leaky." The benefit of stopping retiring some sequence numbers is that the worst case becomes endpoints running out of CIDs - something that is expected to happen on ordinary connections too. Note also that in HTTP/3, reestablishing a connection in 0-RTT is always a viable option. To summarize, I think that the concern can be resolved for HTTP/3 by changing the recommended behavior from resetting the connection to stop sending RCIF frames for some sequence numbers.
NAME I don't think not sending RCID frames is that easy. Tracking something that needs to be sent consumes comparable state to tracking it in flight. Or are you suggesting a different algorithm that indicates a RCID frame doesn't need to be sent? If the peer has to wait for one Retire Prior To to take effect before it's increased again, that leaves only the peer migrating really quickly and lost ACKs as the unbounded case. As NAME said, I think it's more likely a well-behaved peer would run out of CIDs in this case than any other failure mode. I previously suggested stopping sending out NCID frames when the RCID frame limit had been reached to force the issue, but that just changes who has to close the connection. FWIW, we already have sanity checks where our implementation closes the connection when a datastructure becomes too large. The limits are high enough that they're almost never hit unless there is a bug, but they've been helpful in finding bugs. I would advocate everyone have these. NAME You'll never get to 0 transport errors. Some implementations will have a bug, and some peers or servers will have faulty or corrupted memory. For example, in Google QUIC we found we received the ClientHello on a stream besides 1 surprisingly often.
I think that the idea is to reduce the set of RCID frames needed to a range of sequence numbers. You only have to remember the few exceptions for sequence numbers that are in use, sequence numbers for which you have outstanding frames, and a range of sequence numbers that require retirement. As long as your discretionary use is sequential, this is bounded state. As mentioned, the risk is that lack of acknowledgment leads to potentially large delays in getting new connection IDs, but if you ever run short, abandon the connection. Either you have a pathological loss pattern or your peer is messing with you. In either case, it is probably not worth continuing.
NAME Yes, this is one method of limiting the state that is being used for retaining sequence numbers that need to be sent. And it works even when discretionary use is not sequential, as the number of "exceptions" (i.e. gaps) is bounded by maxconnectionidlimit. The other assumption of using this method is that you would somehow bound the state that is used to track RCID frames being inflight. There are various ways of accomplishing that, e.g., have a limit on the number of RCID frames sent at once, track bunch of inflight RCID frames as one meta-frame. And it is correct that the outcome would be a "lag" in this method. The downside of this method is that it might be a disruptive change to existing implementations. It is essentially equivalent to what proposes, with the exception being that tracking RCID frames are difficult (as you track to send many). For existing implementations, it would be easier to adopt a "leaky" method, i.e.: let endpoints have a counter counting on the number of sequence numbers that are retired but are yet to be acknowledged when a CID is being retired, if the value of the counter is below a certain threshold (i.e. 2 maxconnectionidlimit), the counter is incremented, and a RCID frame carrying the sequence number of that CID is added to the outstanding queue. If the counter was not below the threshold, no RCID frame will be sent (leak!) when an RCID frame is deemed lost, it will be resubmitted into the outstanding queue when an RCID frame is acknowledged before deemed lost, the counter is decremented when an RCID frame is late-acked, we do nothing Assuming that the solution we prefer is the least-disruptive approach, I think we need to recommend the latter. If we have distaste in the latter and prefer the former, I think we'd better switch to , as it more cleanly reflects the design of the former. NAME I think the latter is not that a big change to existing implementations.
Yep, I think that is a good approach. Obviously, the range encoding is superior in terms of how much state it can hold, but a simple list is fine. And the effects of lag will be limited by the tolerance for retaining retired connection IDs, which seems proportional to me.
NAME I doubt that's less impactful on existing implementations, certainly from an interoperability standpoint. Today endpoints should hold on to incoming CIDs until they are formally retired. Making them leaky can lead to trapped state. Sending them all, but with a delay, is no added state but solves this problem.
After a lot of back-and-forth, I think that we've all generally converged on .
Still LG, but I like ekr's suggestionLatest text looks good

Related discussion:
text: in connection failures, as the issuing endpoint might be unable to continue using the connection IDs with the active connection. <ins> An endpoint SHOULD limit the number of connection IDs it has retired locally and have not yet been acknowledged.  An endpoint SHOULD allow for sending and tracking a number of RETIRE_CONNECTION_ID frames of at least twice the active_connection_id limit.  An endpoint MUST NOT forget a connection ID without retiring it, though it MAY choose to treat having connection IDs in need of retirement that exceed this limit as a connection error of type CONNECTION_ID_LIMIT_ERROR. Endpoints SHOULD NOT issue updates of the Retire Prior To field before receiving RETIRE_CONNECTION_ID frames that retire all connection IDs indicated by the previous Retire Prior To value. </ins> 5.2. Incoming packets are classified on receipt.  Packets can either be