Change:
NAME NAME As discussed in I added a minimum value of 2 for the TP, since values below this don't make any sense. Having an invalid default value seems wrong to me, so I also made 2 the default value for the TP. People might disagree if this a reasonable value.
I'm seeing some people asking for changes, but I am happy with Marten's responses to those. I'm going to mark the associated issues as .
In , we state that: I think that these are correct. However, we also state: I think this is incorrect. This should instead state that "Connection IDs that are unused and not retired are considered active, ..." That said, I think the name is confusing, as it is awkward to argue that the connection ID that is currently used is "active." Maybe we should rename the parameter to something like "spareconnectionid_limit."
Isn't it impossible to agree on whether a given connection ID was used? In particular, connection IDs sent on unusable paths won't be received by the issuer to allow them to know that they a no longer unused.
I hastily reused the term "unused" from the other text quoted above. So please interpret "unused" as "until receiving a packet with a previously unused connection ID."
FWIW, the alternative approach (that could better match people's intuition) would be to retain the definition of "active connection IDs" as connection IDs that are issued and not retired are considered active, and change the normative texts. Actually, I think that this might be a better choice, due to the following two reasons: Current text requires you to issue new CIDs under two conditions; i.e., when receiving a packet carrying a previously unused CID, or when receiving RETIRECID. We can get rid of the former. Current design has a potential DoS vector that could lead to memory exhaustion on the CID issuer. This is because the number of CIDs that the current text recommends an endpoint to retain is activeconnectionidlimit + numberofCIDsinuse. The number of CIDs in use is something that the consumer of CIDs controls. Changing the number of CIDs to be retained to just the value of activeconnectionid_limit eliminates that attack vector.
NAME I agree, that was my understanding of the spec, since this is what the definition of the TP implies to me: I think it might be worth considering to slightly change the definition of the TP though, such that it includes the CID used during the handshake (and the one provided in the SPA). Otherwise, when the peer provides the requested number N of CIDs, you actually have to store N+1 CIDs, until you retire the CID used during the handshake. Furthermore, on the peer's side, after having provided N connection IDs, it must take care of NOT issuing a replacement for CID 0 when it is retired (whereas all other CIDs need to be replaced).
The implication, of course, is that 0 => infinite, but we don't actually say that anywhere. (This part might be editorial; it omits to say what we all know it means.) 1 is also a mostly-invalid value, since you can't avoid exceeding that limit without the peer dropping to zero CIDs in the interim and therefore stalling the connection for an RTT + any recovery time needed. Also, if the server uses , it has provided two CIDs before it knows the client's limit. This is fine, really, so long as the peer SHOULD NOT exceed the limit; if you send (or default to) a stupid value, the peer will ignore you and all is well. But if we take 's suggestion that exceeding the limit be a MUST NOT, the default needs to be infinite and values less than 2 need to be prohibited.
(Design issue please.) There is a lower point on the idle_timeout, which is . Having 0 imply is awkward, so I think that Mike's suggestion is the best outcome.
NAME I am literally not sure if this is the case. As I've pointed out in URL, the normative language suggests that activeconnectionidlimit actually means the number of "spare" CIDs that should be available, though the definition of "active connection ID" is defined differently. If we are to consider the normative language as being correct, it would be totally valid to set activeconnectionidlimit to 1 (i.e. I want one spare CID!), or 0 (i.e. I do not want any).
The definition of the transport parameter says: So sending a value of 0 (or, as it currently stands, omitting the TP), is a valid thing to do. It's an endpoint declaring that it will never, under no circumstances, change the connection ID, but to keep on using the connection ID provided during the handshake. Of course this also implies that connection migration is effectively disabled. We might want to decide to make this illegal though. Since the introduction of the Retire Prior To field on the NEWCONNECTIONID frame, implementations have the option to retire connection IDs in case their infrastructure requires that. A peer sending a value of 0 would prevent those implementations from rotating their connection ID mapping, which seems like an awkward special case. As NAME points out, sending a value of 1 doesn't make any sense, since you can't switch to a new CID without stalling the connection first. Therefore, I agree with NAME suggestion to forbid values below 2. I'm undecided if we should define a default value, and if so, what that value should be.
Discussed in Singapore; a bit more discussion and then CfC.
The parameters communicates the maximum number of connection IDs sent in NEWCONNECTIONID frames that an endpoint is willing to store. This value is only advisory though, there's nothing stopping an implementation from storing more CIDs than this limit, and the peer is allowed to send more CIDs than this limit: We don't specify any behavior in the case the peer provides more CIDs than this limit, but since retiring a CID asks the peer to provide a new CID, it seems like the reasonable thing to do would be to silently drop the CID on the floor: This means that a peer cannot fulfill its duty of retiring those discarded connection IDs when receiving a NEWCONNECTIONID frame with Retire Prior To set: This, however, is not guaranteed, either. A peer might decide to send a RETIRECONNECTIONID frame for a sequence number that it already forgot. Furthermore, depending on how implementations decide with all these uncertainties, one could end up in a situation where an endpoint only has a single active CID available, but its peer assumes that it is still keeping track of CIDs and thus won't provide any new CIDs. In this situation, the endpoint won't be able to initiate or respond to a connection migration any more, leading to a failure of the connection.
That was the previous recommendation if the peer provided too many CIDs and you didn't want to maintain any more, and I think that still applies if the peer doesn't respect the TP. For context, this came up in several issues, the TP came in via , with the being that it would be advisory only (hence the SHOULD). PR actually made the change which included that SHOULD, and also resolved the concerns from . helped to clarify some rate-limiting around how quickly you should cycle through CIDs. All that is to say, if you want to use the limit and explicitly communicate it with the TP, then you SHOULD do that. Everyone SHOULD do that. If not, or if your peer doesn't respect it, you can always dump those extra CIDs on the floor, with all the potential for running out that comes from taking that action. In this case, you end up maintaining as many as you wanted, probably the same value that you sent in the TP that the other side ignored... If/when the peer sends RetirePriorTo and requests that you retire them, you retire the ones you haven't yet retired and still know about. Generally, that means that you get a replacement for those, so now you still have CIDs. (Side note: The place this gets messy is if the peer suddenly decides to respect the value you put in the TP and doesn't replace them. This has always been a potential consequence of dropping them, though.) For the ones that you discarded, the peer should discard those 3PTO after it sent RetirePriorTo. From : So the peer that requested retirement can discard any state about them, and you're back to having everyone on the same page about how many are outstanding. That generally seems sufficient, although it would be great to cover another scenario or add some text that could be added to explain this better? If the issuing endpoint doesn't use RetirePriorTo then you're fine holding as many as you wanted to maintain forever. If they do use RetirePriorTo, then you're reset back to a good state. And if you ever get RetirePriorTo indicating an old value, you're also happy since you don't need to do anything.
Dropping connection IDs worked out fine before we introduced the Retire Prior To field. We added that field in , which was merged in July (about half a year after our discussion in Tokyo and the claryfing PRs). That's not what the spec says (but maybe what the spec intends to say?). If this was the only issue, we could probably easily resolve this by changing the wording to something like "MUST retire the corresponding connection IDs that it still remembers" or so. That might or might not happen. It all depends on the peer's assumption of what you did: Does it assume that you dropped connection IDs (if so, which ones?)? Does it assume that you actually enforced the limit that you announced in the TP by dropping all CIDs in excess of it? The situation gets even worse when you assume that a RETIRECONNECTIONID frame was in flight just when the endpoint provided the new CIDs. In this situation, there's no race-condition-free view of how many CIDs the peer was actually storing, so it's not possible to know how many of the new CIDs were actually stored and how many of them might have been dropped.
I think the model that NAME has described is accurate. At the same time, it seems alarming to me that we have failed to incorporate that model into the text. Specifically, as NAME points out, we have the following text: and FWIW also: that both assume that all connections IDs are expected to be retired. But as NAME points out, when the issuer provides more than what recommends, those connection IDs are not expected to be retired. I'd like to point out that the way we control the active CIDs sticks out from other "crediting schemes". For all other crediting schemes (i.e. MAXDATA, MAXSTREAMS, MAXSTREAMDATA), there is a hard limit. It is a protocol violation when the sender exceeds the allowed maximum. It is only for this activeconnectionidlimit - RETIRECONNECTIONID pair that we do not enforce the sender adhering the limit. Considering these aspects, and the fact that implementors are complaining about the complexity caused by "SHOULD NOT," I think it's worth considering changing the "SHOULD NOT" of "an endpoint SHOULD NOT provide more connection IDs than the peer’s limit" to "MUST NOT." The counterargument might be that it's an additional complexity; but I doubt if it's that important, compared to the importance of resolving the confusion that we have now. As NAME have stated, the issuer is excepted to stick to providing N active CIDs, where N is a constant throughout the lifetime of a connection. No other approach works. So the complexity that is to be added by changing the "SHOULD NOT" to "MUST NOT" is that the issuer would be required to set N to no greater than the value of "activeconnectionid_limit" provided by the peer. I think we can pay that cost for buying a consistent design (that avoids the confusion).
It seems like the problem we're facing here is that 3 things can happen with a new CID: It is stored for later use. It is retired via a RETIRECONNECTIONID frame. It is silently dropped. For the endpoint providing the CIDs, 1. and 3. are indistinguishable, which might lead to a mismatching view of how many CIDs are actually active. In the extreme case, this mismatch might mean that an endpoint believes that its peer has enough CIDs available, while it actually doesn't have any CIDs besides the one it is currently using, which would then make it impossible to migrate. At the very least, the indistinguishability between 1. and 3. leads to some non-trivial reasoning about what the peer might or might not have done, if you provide more CIDs than the peer advertised at any point in the connection. I agree with NAME that changing the "SHOULD NOT" to a "MUST NOT" would lead to a significant simplification here, since it would eliminate 3. from the list. With that change, a newly provided CID is active until it is retired. Furthermore, I don't think that there's any additional complexity here. On the receiver side, you'll need some logic to protect against a peer that is sending too many CIDs anyway. On the sender side, limiting the number to a constant value seems a lot easier than reasoning about which CIDs might have been dropped.
Changing to "MUST NOT" SGTM
I also agree that changing to a "MUST NOT" seems reasonable.
I think a MUST NOT is reasonable; there was considerable pushback in Tokyo for reasons I don't remember clearly. I don't think you can get out of sync provided the TP is at least 2. The other possibility, as an implementation, is to track which IDs you've retired/forgotten. Obviously, the point of not keeping all the CIDs is to reduce the memory consumption and this still requires keeping some memory. But it's worth pointing out that you don't actually need to have the CID in hand in order to promise you'll never use it again. For that matter, a naive implementation that wants to save memory at the cost of bytes could retire every CID sequence number up to RPT upon receipt of an increase RPT value, whether it has previously retired it or not.
I definitely agree that it could cut down a lot of complexity to change from SHOULD NOT -> MUST NOT, although we probably should check the minutes from Tokyo about why we didn't want that at the time. NAME for: I can see where it might be irritating to not know a precise count here, but I'm not sure the two sides actually need to be in sync about that. NAME point about being able to retire CIDs that you've forgotten is well taken, although it does run the risk that you'll get a replacement (although maybe you drop that too?). I think NAME argument that changing to a MUST NOT would bring us in line with the other crediting schemes makes a lot of sense, and it would bring us to a much nicer interaction model -- you ask for whatever you want, the peer gives that many to you. (Can they still give you fewer? I think the answer has to be yes, if it's costly to generate and maintain, then the issuer should control that.) I think it would drastically simplify if we made this MUST NOT. Happy to send a PR for that. Do we want a protocol violation if exceeded? We will likely need too along with this.
Another potential pitfall of "MUST NOT": Assume you've filled the peer's quota, but now need to revoke all prior CIDs and start a new generation. You have to send at least one NCID to carry the RPT and give the client something in the new generation of CIDs. Depending on the client's implementation, the limit might be exceeded during the processing of the frame. This can be addressed if the client takes actions in this order: Retire all unretired CIDs <= RPT, queuing the RCID frames for transmission Add the new CID to the connection Send the queued RCID frames using the new CID During this transition, though, there are zero active CIDs on the connection. A more savvy implementation might retire all but one of the old CIDs (so data flow never gets stalled), install the new CID, then retire the last old one. Of course, that gets trickier in a multipath world; you have to make sure that the one you keep is okay for the currently-active path. What the client can't do is add the new CID first, check the limit, then process any retirements; in that order, the limit gets breached.
There definitely was a reason we avoided making this a strict requirement, and I think this played into it, but this is the benefit of having RPT delivered in the same frame that delivers a new one. Perhaps modify the text to indicate that the error is if the limit is exceeded after retiring any in RPT? Or even some non-normative text along the lines of “If RPT retires all outstanding CIDS, immediately start sending new packets with the new CID and make sure you only check the limit after retiring any indicated in RPT”? On one hand, it’s kind of an annoying thing to keep track of. But on the other, making it a hard requirement does decently simplify a lot of other scenarios. That said, we should probably go dig up the other reasons we avoided this the first time around.
The aren't super informative.... NAME was against it being an error, and NAME thought it needed to be a hard requirement if present. Nothing else specific about who opposed it or why. I really wish we audio-recorded these meetings. This is why I try to transcribe the discussion rather than just minuting outcomes, chair comments notwithstanding.
Ah, thanks for looking that up! In that case, probably worth thinking through if there are other situations like this, but even if so I'm inclined towards the proposed change, albeit with additional text to cover the edge cases/provide implementation guidance.
NAME IIRC, some mildly opposed to the addition of activeconnectionid_limit because it's an unnecessary complexity, arguing that the model could be simple as: issue-some initially, then issue more as the peer uses the CIDs. After the discussion, the middle ground that we agreed was to make it an advisory settings. Note that we made this decision prior to the addition of Retire Prior To. But now, with the introduction of Retire Prior To, we have another way of retiring CIDs. I think that the simple model is no longer unsustainable.
NAME That matches what I remember from the discussion in Tokyo. I think we actually need normative text along the lines that NAME suggests. I don't think it's a problem that during the time you process a NEWCONNECTIONID frame the number of active connection IDs might temporarily reach 0, if we assume that these frames are processed atomically. And I can't really think of a reason why any implementation might want split processing of the Retire Prior To field and adding the new CID. I updated with a suggestion in 8354804a2a30580fd1b7deadb19f3baf3ad68fd8. WDYT?
I'm going to contradict me-from-yesterday a little, or at least complicate the point. The reason we have the 1-RTO window is that some implementations might not be able to actually retire CIDs instantly (packets are queued in hardware offloads, you want to avoid racing in-flight packets against the packet containing the RCID, etc.). If we require the retirement to be done as part of processing the frame, processing the frame can take up to an RTO. What happens to other frames in the packet? Does this packet get ACK'd if you're not done with this process yet? Plus, you can't actually send the RCID frames until you've got the new CID unless you have a race with one more RCID frame in a subsequent packet. This can get sticky very quickly. If I had an implementation where I couldn't retire CIDs right away, I would probably tag them for retirement upon receipt of this frame and say those that are tagged don't count toward the limit. But that's not entirely compliant with the text in your commit, and also assumes I have some headroom I'm not telling the peer about, because I have to remember all of them temporarily.
I agree that this would be a reasonable thing to do in this case. Clearly, if you have such an implementation, it's a highly optimized one that uses async processing and / or hardware offloads. I don't think it's too much too ask for that such an implementation takes appropriate steps to bridge the time it takes to retire a CID. Do you have any suggestion how to improve the text in the PR without getting too much into implementation details?
Perhaps to say that the peer MUST NOT consider the connection IDs it has been asked to retire when calculating whether the limit has been exceeded, even if they haven't actually retired them yet? That gives implementations the latitude to hang onto them for a bit if needed, but keeps you from going over the limit even so. Then it's the implementation's business whether it wants to defer adding the CID (and block processing of this frame), add it and exceed the limit temporarily, retire the old CIDs on the spot, or something else.
Right, I think the issues about packets that are already enqueued, etc. are more for the person requesting retirement of the CID to know that some packets might still show up (although potentially not after the ACK for the NCID frame that had RPT if they're all in the same queue? Not sure I'd want to rely on that ordering even before looking at the network). The endpoint that is retiring the CID can simply stop sending new packets with that CID and be fine. And leaving it up to the implementation but making a note that it's not a problem to retire/replace right up to the limit sounds good to me.
Proxying from the PR in regards to enforcing MUST error if the limit is exceeded: NAME NAME NAME NAME NAME
Having discussed this offline with NAME and NAME I now understand the new model that you are proposing. I'm fine with it. I'll leave my comments on the PR.
Having chatted about this with NAME and NAME I'm ok with this set of changes. A few suggestions.

Discussion:
available and unused connection IDs.  Endpoints store received connection IDs for future use and advertise the number of connection IDs they are willing to store with the active_connection_id_limit <del> transport parameter.  An endpoint SHOULD NOT provide more connection IDs than the peer's limit. An endpoint SHOULD supply a new connection ID when it receives a packet with a previously unused connection ID or when the peer retires one, unless providing the new connection ID would exceed the peer's limit.  An endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of connection IDs; see reset-token. </del> <ins> transport parameter.  An endpoint MUST NOT provide more connection IDs than the peer's limit.  An endpoint that receives more connection IDs than its advertised active_connection_id_limit MUST close the connection with an error of type CONNECTION_ID_LIMIT_ERROR. An endpoint SHOULD supply a new connection ID when the peer retires a connection ID.  If an endpoint provided fewer connection IDs than the peer's active_connection_id_limit, it MAY supply a new connection ID when it receives a packet with a previously unused connection ID.  An endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of connection IDs; see reset-token. </ins> An endpoint that initiates migration and requires non-zero-length connection IDs SHOULD ensure that the pool of connection IDs

An endpoint can cause its peer to retire connection IDs by sending a NEW_CONNECTION_ID frame with an increased Retire Prior To field. <del> Upon receipt, the peer MUST retire the corresponding connection IDs and send corresponding RETIRE_CONNECTION_ID frames.  Failing to retire the connection IDs within approximately one PTO can cause packets to be delayed, lost, or cause the original endpoint to send a stateless reset in response to a connection ID it can no longer route </del> <ins> Upon receipt, the peer MUST first retire the corresponding connection IDs using RETIRE_CONNECTION_ID frames and then add the newly provided connection ID to the set of active connection IDs.  Failure to retire the connection IDs within approximately one PTO can cause packets to be delayed, lost, or cause the original endpoint to send a stateless reset in response to a connection ID it can no longer route </ins> correctly. An endpoint MAY discard a connection ID for which retirement has been

The payload of an Initial packet includes a CRYPTO frame (or frames) containing a cryptographic handshake message, ACK frames, or both. <del> PADDING and CONNECTION_CLOSE frames are also permitted.  An endpoint that receives an Initial packet containing other frames can either discard the packet as spurious or treat it as a connection error. </del> <ins> PING, PADDING, and CONNECTION_CLOSE frames are also permitted.  An endpoint that receives an Initial packet containing other frames can either discard the packet as spurious or treat it as a connection error. </ins> The first packet sent by a client always includes a CRYPTO frame that contains the start or all of the first cryptographic handshake

0. The payload of this packet contains CRYPTO frames and could contain <del> PADDING, or ACK frames.  Handshake packets MAY contain </del> <ins> PING, PADDING, or ACK frames.  Handshake packets MAY contain </ins> CONNECTION_CLOSE frames.  Endpoints MUST treat receipt of Handshake packets with other frames as a connection error.

length of the Connection ID field. The maximum number of connection IDs from the peer that an <del> endpoint is willing to store.  This value includes only connection IDs sent in NEW_CONNECTION_ID frames.  If this parameter is absent, a default of 0 is assumed. </del> <ins> endpoint is willing to store.  This value includes the connection ID received during the handshake, that received in the preferred_address transport parameter, and those received in NEW_CONNECTION_ID frames.  Unless a zero-length connection ID is being used, the value of the active_connection_id_limit parameter MUST be no less than 2.  If this transport parameter is absent, a default of 2 is assumed.  When a zero-length connection ID is being used, the active_connection_id_limit parameter MUST NOT be sent. </ins> If present, transport parameters that set initial flow control limits (initial_max_stream_data_bidi_local,

for different connection IDs, the endpoint MAY treat that receipt as a connection error of type PROTOCOL_VIOLATION. <del> The Retire Prior To field is a request for the peer to retire all connection IDs with a sequence number less than the specified value. This includes the initial and preferred_address transport parameter connection IDs.  The peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frames in a timely manner. The Retire Prior To field MUST be less than or equal to the Sequence Number field.  Receiving a value greater than the Sequence Number MUST be treated as a connection error of type FRAME_ENCODING_ERROR. </del> <ins> The Retire Prior To field counts connection IDs established during connection setup and the preferred_address transport parameter (see retiring-cids).  The Retire Prior To field MUST be less than or equal to the Sequence Number field.  Receiving a value greater than the Sequence Number MUST be treated as a connection error of type FRAME_ENCODING_ERROR. </ins> Once a sender indicates a Retire Prior To value, smaller values sent in subsequent NEW_CONNECTION_ID frames have no effect.  A receiver

mandatory, was present though it is forbidden, or is otherwise in error. <ins> The number of connection IDs provided by the peer exceeds the advertised active_connection_id_limit. </ins> An endpoint detected an error with protocol compliance that was not covered by more specific error codes.