Change:
Adds a new (optional) field, 'Retire Prior To' (placeholder name), to the NEWCONNECTIONID frame, which allows a sender to request that its peer immediately retire all connection IDs up to a certain sequence number. Once the peer acknowledges this packet, it has at most 3 PTO to follow through or risk the sender closing the connection.
First, I don't see the frame type rearrange as a very large change at all. Second, I do think the atomic assurance is important, and it's just an oversight that we aren't explicitly require Sequence Number to be greater. I didn't think of that scenario. Without the atomic assurance, I worry about the scenario where an endpoint retires all the CIDs without replacing any. It gets a lot more complicated if you have separate frames. What conditions are considered protocol violations? In order to replace all existing CIDs, the order of the frames in a single packet starts to make a difference. You have to make sure you have the NEWCONNECTIONID frame followed by the RETIREALLCONNECTIONSIDSWITHSEQUENCENUMBER_BEFORE frame.
I think I agree with NAME that we should refrain from renumbering frames (again). One of the issues regarding the proposed renumbering is that the two CONNECTIONCLOSE frame types would not have the same MSB bits; it could be a concern for certain stacks. OTOH, it is also my view that we do not need a new frame nor a new frame type. How about just making the "Retire Prior To" a mandatory field of the NEWCONNECTIONID frame? The value can be set to zero when retirement is not necessary. We do not expect a lot of NEWCONNECTION_ID to be inflight; spending one additional byte is IMO a non-issue.
+1 to NAME suggestion of making it mandatory and not renumbering frame types.
Thanks NAME NAME NAME for your feedback. I have made some changes, making the Retire Prior To field required, so we only have one frame again, added more 2119 language, cleaned up the text on ignoring reductions in the Retire Prior To field, and made it a protocol violation to have Retire Prior To greater than or equal to Sequence Number. NAME on your request to require the difference between the Retire Prior To field and Sequence Number field to be no greater than activeconnectionid_limit, that would mean the sender has to keep track of both what's the highest retired sequence number retired by the peer and what it wants locally, and the max of those two values in this field. I'm not sure the extra complexity for keep track of that is worth the cost. What benefit do you see in that requirement? NAME on the topic of the handshake and preferred address CIDs, I thought we already had text that implicitly assigned them sequence numbers. Is that not the case? What text beyond that would you like me to add?
Thanks for writing this up! Specific to this text (see additional discussion on the issue): I don't think that we need any requirement about the distance between RetirePriorTo/SequenceNumber activeconnectionid_limit. Setting a super high value would be a bit nonsensical, so some non-normative text explaining why doing that is a complete waste of everyone's time would be good, but it doesn't actually break anything. Saying "hey this is silly, consider that limit when you set this" is probably good enough just to make implementors aware of the foot gun. To the point about handshake and preferred address CIDs, I think it would be nice to have some text just as a reminder along the lines of "don't forget these have sequence numbers too" to help folks avoid confusion.
I agree that atomicity is important here, if we want to avoid the situation where all connections IDs are retired at the same time. Adding the field to the NEWCONNECTIONID frame as NAME suggested seems to be the most straightforward way to do so. I'm not a big fan of the 3 PTO requirement here. I don't see why it shouldn't be possible to retire the connections immediately when receiving the frame. There's a DoS risk here: if an attacker delays a packet with a connection ID that was retired afterwards, it can cause either a stateless reset or even a PROTOCOLVIOLATION. The better way to enforce the retiring of the connection IDs is to look at packet numbers: The old connection IDs MUST NOT be used in packets with higher packet numbers than the packet that acknowledged the NEWCONNECTION_ID frame.
NAME Am I correct in understanding that this is the way key update works? Assuming that is the case, my +1 goes here.
I think that's an implementation decision. You could certainly enforce this by requiring that . This doesn't stop a peer from acknowledging the NEWCONNECTIONID frame, but never acting on it though. Another way to implement this would be to enforce that . This would be a somewhat stricter way, since it doesn't allow the peer to ignore the frame (without inducing artificial packet loss).
I think this solves the problem since I really dislike that a delayed packet, 3PTO or otherwise, can take down a connection. However, I also dislike that this is coupled strongly to ACK behaviour. I'm not sure how key updates work but assuming it does not use ACK, ACK handling can run asynchronously in a separate task, and indeed this is how I would architect a high performance implementation. Then one could send a dedicated frame to acknowledge receiving the retire frame, but that is pointless since any packet using the new CID would have that effect. So for the time being I don't have a good answer to this. Altogether, if this facility opens up to DoS attacks and prevents ACK handling from running asynchronously, I think it does more harm than good and should at least be moved to v2 giving more time to consider the implications.
NAME the problem is that the sequence can be sparse. The client isn't required to use the CIDs in order any more, IIRC; so you could have four outstanding CIDs, but your four are sequence numbers (2, 127, 324, 325). Nick wants to expire everything prior to 300 and he's tired of waiting for you to use and discard the two early ones.
NAME I'm all for making it immediate and adding stronger language. That's exactly what I wanted to do in the first place. Initially, I was getting push back though, so I relaxed it a bit. I agree it's a bit too wishy-washy in its current form. Unless anyone has reasons why they can't or shouldn't update immediately I will go ahead and strengthen the text.
I think the main reason not to be totally immediate is when you have some distributed processing generating packets. You might not be able to guarantee that every thread/process generating packets can stop using those CIDs immediately on receipt of the signal that the issuer wants them gone. However, we already went through that with the RETIRE_CID discussion -- sending RETIRE is supposed to be the signal that you've hit your sync point and will not use them any more.
Brainstorming times when it would be valuable to not change immediately: Privacy, a client that wants to avoid linkability on different links shouldn't have to change all of its CIDs regardless of link immediately, since that would trivially link those different CIDs If we have a server behind a load balancer and it wants to rotate keys, having every client connected to that individual server change CIDs right away could betray a bunch of information I don't know if we need to go as far as recommending that clients wait some amount of time, but we could potentially (in QUIC LB maybe?) recommend that servers rotate their clients not all at once. And clients might want to change on different interfaces at different times. I do like the idea of having a packet number cutoff when the client retires the CID to simplify things on the other end.
IIRC, the conclusion we reached on the CID design team was that: When the client sends RETIRE, that's a statement that it has purged that CID and SRT; the CID will never be used again, and the SRT won't be recognized. The server can immediately declare that CID void upon receipt. If an earlier packet got delayed, it will trigger a SR, because the server doesn't recognize the CID any more. This is also true if the client messed up and sends a later packet with the retired CID; the server will discard or reset it. The client will receive the server's reset, but not recognize it as such because the SRT isn't one that it knows any more. The important thing is that the client has already deleted the SRT before sending the RETIRE and the server can safely purge state immediately upon receipt. In this PR, I think we have the right model by letting the client have a little time to make sure those requirements are true before sending the RETIRE. You MUST retire the CIDs, but you're not obligated to do so instantly.
In line of NAME comments and what I have said earlier: Implicit retirement of "Prior To" has the issues mentioned with distributed ACK processing. But the problem is in fact worse: For almost the same reason that the sender wants to retired CID's, the receiver may want to be either migrating internally to another host, or process different parts of a connection on different servers. It can be detrimental to such deployments to have a force expiration. Therefore I think Mike is right in tying this to RETIRECID. Perhaps the right model is to use RETIRECID as the sync point regardless of whether this is done voluntarily or motivated by a "Prior To" flush. Mike suggests the sender should defer RETIRECID until safe, but I think the receiver should make that call because it might have a 10 minute periodic load balancer flush or something completely different while the sender can only estimate based on RTT and the sender also wants to forget the CID ASAP while the receiver can track CID's algorithmically. The recommendation could be to invalidate CIDs from RETIRECID frames no earlier than 3PTO and possibly longer if it suits the deployment. Receiving a non-zero "Prior To" field SHOULD cause the CIDs to be flushed and retired with RETIRECID as soon as practical. The issuer of "Prior To" MAY close a connection with the error CONNECTIONID_TIMEOUT but deployments SHOULD avoid doing that aggressively unless critical to infrastructure since clients may not be able or willing to updatee CIDs fast. Note that there can be race conditions where both endpoints want to retire CID's because they are internally migrating. This can lead to problems if a CID is tied to the peers CID indirectly via source IP. We chose to have asymmetrical path migration due the complexities of this. I start to think we could some nasty corner cases here.
I'm OK with "SHOULD immediately" but I still think having an endpoint close the connection with protocol violation if a packet is delayed is a security risk for the reasons I detailed in the issue.
Personally, I don't think there is a big difference between explicitly closing the connection VS just removing the CIDs from the routing lookup, which would cause a stateless reset to be sent if the peer continues to use it. I like the explicit nature of the close, because it will be easier to debug if there are bugs; though I do concede that if the peer did locally retire the CIDs, but somehow the packets/frames just took an extreme amount of time to get there, the connection isn't penalized. Perhaps a vote? Thumbs up if you prefer connection close with protocol violation. Thumbs down if you prefer the CIDs to be removed, resulting in a stateless reset instead.
NAME As NAME has pointed out in URL, "having the use of a CID you've asked to retire trigger a protocol violation is a security risk. It allows an attacker to close the connection by delaying a valid packet." If it is the case that we would have the same issue in sending a STATELESS_RESET (I assume that is what you are suggesting), I think we need to fix the issue, rather than choosing one between the two designs that both have the same security concern.
PS. One way of resolving the security concern would be to require the use of Stateless Reset (instead of a CONNECTION_CLOSE), and also require that Stateless Reset Tokens for CID above "Retire Prior To" to be different from the tokens below or equal to "Retire Prior To." Note: the latter requirement is necessary because states that "the same Stateless Reset Token MAY be used for multiple connection IDs on the same connection."
I agree with NAME If we want to close the connection (which I still think we shouldn't mandate but I think we can allow) then we should use stateless resets, mandate each CID has a distinct reset token, and mandate that endpoints ignore reset tokens for CIDs they have retired. That'll prevent an attacker from closing the connection by delaying a packet.
That goes back to my suggestion of using epochs ("prior to" being the simplest such) because you might not want to store a reset token for every CID, only for every epoch.
I agree with NAME and NAME NAME In general, each CID is associated with its own reset token. This allows for a straightforward stateless implementation of a stateless reset token generator. I don't think it would be a good idea to revisit that principle.
NAME no you are probably right.
Thanks. All your arguments make sense. I have updated the text to say the connections IDs should just be removed after the 3 PTO, instead of closing the connection. I also added some text warning against reuse of the stateless reset tokens.
I'm not sure I know what "remove the connection ID" means on the receiver. I would remove any mention of 3 PTO as well. Simply state that if a sender keeps using a CID that its peer has asked to retire then packets can get misrouted or dropped.
NAME by "remove the connection ID" I meant that the connection IDs would be implicitly treated as retired. I'm not sure how best to word that section of the text. Suggestions are welcome. Additionally, I don't think many folks (though I'm not among them) would like the removal of the 3 PTO text. Several people have argued for allowing continued use of the CIDs for a short while.
I think the text needs to be more clear on whether RETIRE CID must be send for those requested to be retired, or if that is implicit.
NAME we have the following text already: Is there something else you'd like added?
"The peer SHOULD stop using all the connection IDs as soon as possible and then retire the connection IDs by stop sending a RETIRECONNECTIONID frame for each connection ID." "retire immediately" is a bit too strong as several have commented.
This is true, but the text currently allows for the SRTs associated with various CIDs to be identical. This becomes problematic if you're forcing retirement and want to stop recognizing some of the CIDs previously issued. However, that's a problem we already have and cover in the text: This text seems to effectively exclude silently dropping some CIDs that share a SRT with a currently-active one. It might be worth mentioning that explicitly, if we recommend a silent forget approach.
This is looking good (modulo the one extra paragraph at the end)!
I think the protocol should allow for the originator of a CID to retire it and once that packet has been acknowledged by the peer, remove it from its lookup table. I have a scenario where I want to move a connection from one "partition" to another "partition" on the same machine. I use some bits from the CID to indicate the partition ID. My problem is, I've already sent out a CID to the peer indicating the old partition ID. If I change, I'm going to get a cross-partition lookup perf-hit for the rest of time on this connection. So, I'd like to be able to pick a new partition ID, generate a new CID and send out a NEWCONNID and a RETIRECONNID for the old CID. Once the peer acknowledges this, I remove the old CID from my look up. With this design, if I need to change partitions for any reason, I only pay a perf penalty for 1 or 2 RTT. Another option instead of supporting RETIRE for my own, could be a REPLACECONNID [orignalsequencenumber]. That might be a little less disruptive for implementations to manage. Just copy over the old one with the new one and go about your business.
Would URL meet your needs, other than the fact that it's an extension rather than core protocol functionality?
NAME it might but I don't like the idea that it's a request. If the peer doesn't do this, it is causing two partitions to have cross-partition access for an extended period of time. I don't want the peer to have that type of control. With the extension, even if it claims support, what's stopping it from not actually implementing it? If the act of acknowledging the packet allows me to drop the CID, then they have a great incentive to do it properly. Otherwise the connection dies. And finally, yes, I want it in the spec. Not optional. Back to the whole problem of I don't want the peer to control my perf.
I'd be happy with adding a "If it's acknowledged and a packet with an equal or larger packet number uses the CID, you MAY close the connection." ie: A receiver MUST honor it, the sender MAY enforce it.
That would definitely make things better, but only if they actually supported the extension.
This goes back to a lot of the core concept around vending a CID and having that be a commitment that you're willing for the other side to use it. If we're going to add a bunch of extra machinery around it, maybe it's worth reopening the original conversation? Alternately, it also seems like a lot of these issues would be resolved by a "here's a bunch of new ones, please kill all of the old ones", since in almost all of these situations we're saying "I encoded something in the old ones that is no longer valid or helpful".
NAME that's why I kind of like the idea of REPLACECONNID. Just replace the bytes from whatever CID was previously shared.
The mechanism for REPLACECONNID is not complicated, but I'm not yet convinced about its need. So, if we treat the two partitions as two servers, a commonly used strategy for migrating load is to send a GOAWAY to the client and have new connections be routed to the new server. Is this not adequate for you, NAME
The partition can be on the same physical machine (different processor/numa node). Changing the partition ID allows for dynamic rebalancing of load across these more easily.
I'm a bit torn here - I don't think internal process optimisation should be on the wire, but with 96 cores and counting, and possibly hardware optimized queue delivery, there is probably a reasonable case for it. There may also be a case of handover on the internal network. But this is only a use case consideration. I haven't considered whether NAME proposal is the right way to do it.
Allowing for an endpoint to edit/replace existing CIDs can allow all sorts of dynamic rebalancing logic on the LB side of things. Perhaps server ID is changed for a back end server and it wants to update all CIDs to encode the new ID appropriately. The current design of give something to the peer and you're stuck with it forever, until they decide to stop using it seems overly restrictive. Additionally, if we support inplace CID replacement, the actual implementation logic to support this should be extremely trivial; just memcpy over the old one with the new one for the most part. The complexity only comes from endpoints that want to actually replace their own CIDs. They would need to have synchronization logic for it and so on. Endpoints that never need to do this don't accrue any additional cost.
Even if you do a REPLACE for future IDs, you can't currently require that the client stop using the current CID. So given that you have to wait an arbitrary amount of time for connections to "drain" anyways, I'll go back to my earlier comment: why not send GOAWAYs for connections you want to shed to the new partition? If you really want to have dynamic balancing, it seems to me that there'd be more to gain by forcing the client to stop using the existing CID and using a new one. You would want to REPLACE IDs as well, but without forcing the client to move off the current CID, most connections won't move since most connections are unlikely to use a new CID unless the client is migrating (an uncommon event).
Why can't I require the client update the CIDs? As discussed earlier in the comments, the design would be that the client MUST update and if the server receives an ACK for a packet with the REPLACE_CID and is still using an old CID, it will terminate the connection. HTTP/3 is not my only QUIC protocol, but either way, I want a QUIC solution, not a higher layer protocol solution since QUIC is the layer doing this load balancing.
NAME I think your use case is real, and the implementation work client side seems quite small, but it is one more CID feature and I'm not sure what fraction of implementations want to use this? I can imagine we'd use it on the server side if it was present, though.
As I've pointed out, I don't think the mechanism you're asking for is adequately useful for the problem you're trying to solve. I'm not opposed to solving the problem, which is why I'm trying to understand the problem itself. As I said, you would still want a mechanism to replace the existing CID, without which you are stuck draining open connections that aren't changing CID through the life of the connection. I expect this to be fairly common -- most clients are unlikely to change destination CIDs unless there's migration involved. Given this, I still don't see how simply replacing future CIDs is adequate. NAME : Do you think the proposed mechanism, which does not replace the current CID, is useful? Where do you think you might use it?
There is also the concern of conflating migration with targeted load balancing. Even if you did replace the current CID, what happens if the client migrates frequently. All future CID's would be drawn from a pool of CID's that might point to the wrong endpoint process. This means you would have to split up the CID's into two parts: an address and a sequence number, both appearing random externally. You would then change the address part, but leave the sequence selection to the client. This appears to be rather overengineered. Alternatively the current CID could be replaced post migration which might just be few roundtrips of sub-optimal performance before hitting the proper destination, under the assumption that migration is is not that frequent. This also shows that having the server initiate a migration (which is not currently possibly) is insufficient unless the client future CID's are tied to a specific server path.
NAME The mechanism would have to include a way to request/force the client to change CID ASAP to be useful. Only a way to retire outstanding CIDs is fairly useless in my opinion. I assume the mechanism to enforce this would be a GOAWAY followed by a connection close containing a FAILEDTOROTATE_CID error if the client didn't change in time.
Discussed in London; reservations about whether this is desirable / necessary, but NAME to make a proposal. Atomic nature a particular concern, as is the mandatory nature of this.
REPLACECONNID sounds simple, though we need to remember that this frame might arrive out-of-order. That means that this frame needs to have it's own sequence number and the receiver needs to retain the largest sequence number for each of the Connection ID that it maintains...
REPLACECONNID is interesting, but we need to go through the rest of the issues surrounding changing that base assumption. It's potentially worth working through the fallout from doing that and seeing how icky it gets.
I think we should not do this as this stage, thinking of how messy it was to evolve path migration. Also multi-path might complicate things further. Suggest v2.
I agree, that because of reordering, we can't require immediate usage of the new CID after the packet/frame is acknowledged. Instead, perhaps a 1-RTT timer, after the REPLACECONNID is acknowledged, then I just remove the old CID from my routing table, which means those packets would then be dropped. On the side that receives the REPLACECONNID, it would have to ensure that it replaces immediately though, because if it continues to use the old CID after ACK'ing will eventually get a stateless reset. I have several performance problems I am trying to solve with this right now, which I am trying to push a solution in v1. If the solution to this is ever optional (different version or extension) then it's not going to be a valid solution, I would have to go back to the drawing board.
I started writing up a PR for REPLACECONNID and I realized that the frame would end up being pretty much exactly that the same as NEWCONNID. So, one way we could support REPLACECONNID semantics would be a smaller change to allow for NEWCONNID to overwrite a CID sequence number. Practically that's exactly what REPLACECONNID would do, but adds a new frame and a good bit of new text. So, would folks rather change NEWCONNID to allow for overwriting, or add a completely separate frame?
I am not sure if that is the case. Consider the case where the sender first decides to replace CID at sequence N with value X, then decides to replace that CID with value Y. I think we need a way to distinguish if which of replace-to-X replace-to-Y was sent later. So, we need a sequence number for each of the CID that is being replaced.
If it only takes effect if the CID is still current it avoids a lookup to scan all queued CIDs. This would also address NAME concern I believe.
NAME It doesn't, because NEWCONNECTIONID specifies the CID being replaced by using the sequence number, not the current value of the CID being assigned for that sequence number.
Yes, but there is a proposal to allow for a special case where the CID is used explicitly. I'm not sure that would cleanly map to the same frame given that, as you say, the other CIDs use numbers. Also, it wouldn't allow a global reroute if a process moves. All other pending CIDs might be void. Hence a flush all CIDs if current CID match might be needed.
REPLACECONNID would have the exact same reorder problem. One solution is that the sender must not update a connection ID until the previous packet/frame was acknowledged. The other possibility would that receiver keep track of the packet number the connection ID was received in. Always use the CID in the most recent packet number.
I’d really like to avoid doing any of these. This would create a lot of additional complexity.
I'd prefer to have the sender have the responsibility, and in that case, only implementations that actually want to change their CID gets the extra complexity.
I do not think that either of the two would work considering the fact that an endpoint is permitted to retransmit out-of-date frames as they are. Consider the case where an endpoint first sends packet(pn=x,REPLACECONNCID(seq=y,newvalue=a)), then sends packet(pn=y,REPLACECONNID(seq=y,newvalue=b)). If pn=x is deemed lost, then the endpoint is permitted to send(pn=z, REPLACECONNCID(seq=y,new_value=a)), where z is greater than y.
Again, if the responsibility is on the sender side, it's up to the sender on how to enforce it. If you support the feature, you cannot blindly retransmit these frames.
After talking to a few folks, I'm now leaning towards the following solution: Update NEWCONNID frame to have two frame types (or we can add a new, separate frame) where the second frame type indicates the presence of an additional field: (we can bike shed the name). This new field indicates the sequence number of a CID that is being replaced by this new CID (which has a new sequence number too). The sender sends this frame for any/all outstanding, non-retired CIDs that is needs to replace. The receiver is expected to quickly, if not immediately respond with a RETIRECONNID for all the CIDs that the sender is replacing. Once the NEWCONNID frame is ACKed the sender may drop all the retired CIDs from it's routing table after some time (3 RTT?) even if the peer hasn't retired them; OR it can send a HTTP goaway. One note on the CID flow control: Even if the maximum number of CIDs have been given to the peer, an endpoint is allowed to retire one or all of them with this NEWCONNID frame, and the peer is expected to accept these new CIDs; since they are replacing the old ones.
This seems sensible. Why not always specify RetireSequenceNumber? This is nice because it further incentivizes peers to provide them in order, since they're retired in order. If we wanted, this would make enforcing the active connection ID limits trivial(NAME ), because a receiver of NEWCONNECTIONID would compare the incoming ID to the RetireSequenceNumber and if the gap is larger than activeconnectionid_limit, close the connection.
I think the ACK optimisation should be avoided. A CID is retired when it is retired, not before. This also allows peers to opt in to their own schedule. They might have a complex distributed pipeline that is not synced with ACK logic. It is the senders headache since it is trying to force its own optimisation onto the peer. While I can see the overall idea, I'm a bit concerned about race conditions here. If you retire 5 CID's the might not all take effect immediately so when the first one is retired, the next available CID is used even if it is scheduled to retire. Forcing the peer to sync on this requires processing multiple frames concurrently assuming they all land in the same packet for a start.
My primary goal of this is to force the peer to change. I'm OK if it takes a little time, but I need them to change. If they don't I will kill the connection. If we don't have any enforcement, then peers have no reason not to just ignore this and effectively nullifies the feature.
You can get around the race condition by assigning an epoch to CIDs and retire an entire epoch at once. Obviously you'd need to prime with new CID's on a new epoch first, but you can do that in the same frame as the one retiring an epoch. This might also work better on multi-path going forward.
Thanks for writing this up, NAME I think this is going in a good direction. Ideally, RetireSequenceNumber would either mean "SHOULD retire as soon as possible" or "MUST retire", we can debate that as necessary. To NAME comment, there still needs to be a way to increase the number of CIDs that are active (especially at the beginning of a connection), but I think having the field always be present would be fine. I think the key idea here is that, by asking the other side to retire the CIDs, we can manage to avoid any ACK-tracking or other draining scheme. Instead, we request the peer to use the existing mechanism to remove the old CIDs and all of the logic currently in place for that just works for this too. Instead of asking the sender to track the ACK for NEWCONNID, I'd rather have it just wait for the CID to be retired, and if it becomes untenable to do that, it can close the connection (either with no error, or we can define an error code saying "you didn't retire, please connect again").
Yes, I think that is fine. It is just hard to place a hard limit on that.
Why does the replacement need to be coupled to the retirement request? Why not say "please retire all with seqno lower (or equal to) X", then provide a new connection ID?
NAME I presume you mean NAME proposal (URL). I think that can be considered simpler to the RETIRECONNID proposal, because "retire all with seqno lower than X" frame removes the need to retain per-CID sequence numbers.
NAME my main problem with that design is the CID flow control. If an endpoint has currently maxed out the peers max number of CIDs, and it wants to replace them all, what then? If you don't have a single frame to both retire and replace, then it looks like we have a problem. You obviously can't just retire all outstanding CIDs. Then they have no CIDs to use. You also can't just give them more CIDs first, because the limit has been reached, so they will ignore the new ones.
This is one way to implement the epoch I was referring to in the above.
NAME you can have two epochs: old and new each with a limit. This allows you to replenish CID's in new epoch before retiring the old.
NAME I think I'd prefer to have a "please retire" come with a replacement since otherwise there's not any guarantee that they'll be replaced. Having them in the same frame also makes it easy to replace ones without depending on ordering between frames of a "retire" and then "add new one" if you're already at the limit (as NAME said).
There is never any guarantee that the connection ID will be replaced. Coupling them just complicates the design. The idea that you can provide replacements without blowing the limit is strange. You are still providing extra connection IDs. For the case that NAME cares about, the low water mark is probably fine because you will want to send the retirement before filling up the active connection ID set. I think that it is enough to provide a retirement signal and then wait for the peer to retire old ones and replace them on a regular cadence. Worst case, that means that you have to wait a round trip; but that's not going to be a problem for the RSS scenario Nick described.
You're providing an extra ID and have the ability to retire at the same time, so you can easily avoid blowing the limit. This might actually simplify my implementation, because I could store a fixed number of a CIDs in a ringbuffer of the size I specify instead of a hashmap.
I also think the simplest solution to implement is the NEWCONNID frame that replaces an existing CID. NAME if the is always there, what do you set to it when adding a new CID that isn't actually replacing anything?
Set RetireSequenceNumber to 0?
Is that not the first CID exchanged during the handshake? We need to be able to replace that one.
We could make the semantics: Retire CIDs under this number?
I think I prefer the semantics of retiring everything up to a stated offset. That avoids messing with sequence numbers of sequence numbers, which just seems cleaner. The field needn't be always present; the value can only advance, so if you've ever seen a value, you need to retire everything below that value. Corner case to consider: What if I've received CIDs 1,2,3,5,8 and I get a =7? Do I need to retire 4 and 6 that haven't arrived yet, or do I need to retire them upon receipt?
NAME I think you need to retire 4 and 6 immediately. Overall, I think this proposal is nice. At the cost of a single varint, we achieve clearer connection ID management semantics and the ability to retire CIDs proactively. I can see this being pretty useful. One note is that this means a client really has to support changing CIDs, whereas at the moment a client could support it during the handshake and never again. I don't think that's overly onerous, but it's worth pointing out and may be a pro or a con depending upon one's perspective.
I agree with NAME that you need to retire everything (and practically, remember the limit in case you get a heavily reordered packet offering the old one later). If folks are happy with this design, I can write up a PR.
Thanks for writing up NAME I think either of the following two approaches could be a good path forward (discussing here rather than on the PR): The current PR, each new connection ID comes with a "retire CIDs before sequence number X", which keeps us from having issues around holes in the set of valid sequence numbers for a CID. This is kind of nice, because we can be a little looser with how we handle being at the max number of CIDs and needing to send some more/add new ones. Example: I want to retire all the CIDs that are outstanding. I've issued 1, 2, 3, 4 in NCID frames and we've got the handshake at 0. I send a single packet with: Note that sending for all could be specified as valid or invalid, although if valid it doesn't really help with requiring replacement. We'd also then want to consider writing a bunch of other somewhat messy text to require retiring them in order from oldest to newest, which seems somewhat icky. A new frame for just the retire side, called something like . This would be a request, and an endpoint receiving it SHOULD retire immediately, along with a warning that the other endpoint may choose to close the connection if it doesn't. The endpoint also SHOULD move to a new CID if the one it's currently using is requested to be retired. It's probably worth being a little bit careful about how strongly we make not moving a protocol violation, because it seems like it wouldn't be too difficult for an endpoint whose peer had to migrate off the one it was currently using to end up receiving packets from the old CID for a good while/draining period. If we do (2), in order to have the guarantee that they're replaced, we should just include that all frames MUST also be accompanied by a frame, and have the receiver close the connection if that's not the case. (Timing/[not] keeping things in the same packet strikes me as a part of the text here that needs to be written carefully.) Same example from above: In my mind, the common text between both approaches is: The endpoint being asked to retire CIDs SHOULD retire them immediately, and move to a new CID if necessary. If not, the other end MAY close the connection (plus note about not being too aggressive here to handle draining). The endpoint asking to retire them MUST replace them. If not, the other end closes. How we handle "bad" sequence numbers being sent (too low, also a little weird to send a number higher than the peer's limit away from what's active). How you handle when you're at the max specified by the peer's limit Any preferred address and handshake CIDs should probably explicitly be called out as affected, although they do have sequence numbers If someone requests to retire CIDs that you haven't seen yet, you respond to frames containing them by immediately retiring them. This doesn't add too much state since you just need to keep a single counter of the highest invalid/lowest valid sequence number.
I don’t think we should have dependencies on other frames, then I’d rather have a single frame. It would mess up retransmission logic etc. On 5 June 2019 at 19.21.11, Eric Kinnear (EMAIL) wrote: If we do (2), in order to have the guarantee that they're replaced, we should just include that allPLEASERETIRECONNECTIONID frames MUST also be accompanied by a NEWCONNECTION_ID frame, and have the receiver close the connection if that's not the case. (Timing/[not] keeping things in the same packet strikes me as a part of the text here that needs to be written carefully.)
NAME why couldn't we make it a "MUST immediately retire" instead of a SHOULD? What reasons would an endpoint have to ACK the packets but not retire the CIDs at the same time?
NAME in some implementations the ACK logic is not next to the connection ID management logic. For some of these implementations it would be very hard to ensure that the ACK isn't sent with the CID you'd like to retire. "MUST immediately retire" would add significant complexity to those implementations for very limited benefit. Also, having the use of a CID you've asked to retire trigger a protocol violation is a security risk. It allows an attacker to close the connection by delaying a valid packet. I like the idea of adding a field in NEWCONNECTIONID frames, but we should make it advisory and not mandatory. Sending it tells the peer that if they keep sending with this CID packets might not get through at some point. But I don't think we should have MUSTs, immediatelys or protocol violations involved. Is there any reason that wouldn't serve the needs of your use case?
I argued about the same as NAME in the associated PR.
I'm going to try to answer all the comments/questions from both the Issue and the PR here, so as to not clutter up the PR. It now seems that most people actually prefer the immediate switch of the CIDs instead of allowing for delay. I agree key update is a good example / pattern to follow here. You get a packet with the NEWCONNECTIONID frame to replace existing CIDs, you locally update all CIDs and then you mark the packet for ACK. NAME I understand that some implementations might have extra difficulties, but I'm not sure why it would be that much different from key update. Additionally, you only ACK a packet after it has been fully processed, so if you process the NEWCONNECTIONID frame before the "packet fully processed state" and update CIDs there, then there shouldn't be an order of operations problem.
As NAME said: Previously this had been proposed as a way to think about it, and I think it's probably the right balance to strike. You don't work to hurt the client if they don't switch right away, they may have reasons for a slight delay there. A client knows that the request from the other side serves as fair warning that the CID may not work indefinitely. It's still up to the client to use RETIRECONNECTIONID and all of the draining logic there still applies -- we don't need to reinvent additional signals for that.
Immediate is a little strong. There are valid implementation reasons that an "immediate" switch might not be possible -- or at least impossible to guarantee. The immediacy guarantees are that when the implementation sends RETIRE, it has completed whatever is needed to ensure everything is done. An ACK doesn't logically carry the same guarantees -- it just guarantees that the consumer knows about the request. It can be mandatory for the client to retire the CID, enforced by permitting the issuer to kill the connection if it's not done soon enough, but mandatory immediate retirement probably doesn't work.
Just to pile on here. If there is an implementation that is distributed, with multiple entities producing packets (which is entirely possible with some coordination), then retirement requests can take a little time to propagate. I think that the ACK here should be taken to mean, as NAME says, I got your request, not coupled so tightly to compliance. I don't believe that we have any other such things in the transport. STOPSENDING is perhaps the closest analogue, but that doesn't require that you send a RESETSTREAM in the packet that acknowledges the STOP_SENDING.
I don't have a strong opinion on this, but the new discard keys PR that landed has that shape: URL Later packet numbers not using a given key is fairly similar to later packet numbers not using a given connection ID. Enforcement seems comparably or slightly more difficult, so one could go with MUST retire/MAY close the connection. NAME Once you're talking about distributed peers, I think we're well in the space of multi-machine multipath? Or is there some other use case you have in mind?
NAME I think this is a little bit different, since we have a mechanism to retire CIDs and it this is a request to use that mechanism, not an implicit retirement of the CID? It seems like it would be fine to require that after a CID is retired that later packet numbers not use it (I think that's already a requirement?), but doing anything based on the ACK of the "please retire" message doesn't seem like the right path forward here.
NAME we have strengthened the text, so that it's no longer a request, but a requirement that the peer retire the CID. I also agree with NAME that this is very similar to key update, and if an implementation can synchronize keys and packet numbers, I see no reason why CIDs and packet numbers also cannot be synchronized. Practically, for every packet you send, you pick a CID, a key and a packet number to use at about the same time. The only scenario that I've heard that gives me some pause is if the peer is currently trying to probe on a new path when you are trying to force the CIDs to be retired. It might be easier for the probing logic to continue using the CID it started the probe with. But I'm still not completely convinced. For one, what if that new path has a MUCH bigger RTT than the old one? That would make it a lot more likely it could hit the 3 PTO timer on the primary path. Also, I see no reason why you couldn't immediately switch the CID used for probing as well.
I'm fine with the "please retire" semantics, but wanted to point out an existing case in the QUIC specs we say you must retire something in a similar circumstance. I hate timers, so I suspect my implementation will do that even if the spec doesn't require it, because I believe it's more straightforward to implement. If you supply a CID with sequence number X and I've specified that I'm only going to store Y sequence numbers via activeconnectionid_limit, can you assume the peer is unlikely/unwilling to use CIDs with sequence number < X-Y(modulo reordering). That would be true for my implementation, but possibly it's also related to the 3PTO issue?
I don't think this is analogous to key updates. Due to the design of key updates, when you receive a packet with a new key phase, you need to install the new keys, otherwise you just can't decrypt it. The requirement to start sending using those new keys is not much of a stretch. Here, when you ask me to retire a CID, there's nothing I actually need to do to keep parsing packets. Asking the peer to retire a CID is closer to STOP_SENDING than it is to key updates. That said, our implementation will probably retire the CID immediately, but I really do not want text in the spec about closing the connection after some timer, because that opens up new avenues for attack.
NAME what attacks did you have in mind? Obviously, our implementation should not do that if we think it creates new attacks.
With the current text, we aren't explicitly closing the connection, but just implicitly retiring the CIDs after some time, if the peer doesn't do it themselves. So long as the new CIDs have different stateless reset tokens than the CIDs they replaced, I don't see any way for an attacker to cause the connection to be killed.
NAME Consider a MITM attacker that can intercept packets (prevent them from arriving now, and save them for replay later). Assume the server says "please retire CID X", the client switches from CID X to CID Y, and the attacker intercepts the last packet the client sent with CID X. Then the attacker waits 4 PTO and sends that packet to the server. If the server used the same reset token for both CID X and CID Y, then the server will send a stateless reset and the client will close the connection upon receiving it. There is no text in the spec today that says the server MUST pick different stateless reset tokens for different CIDs. Therefore, the ability to request to retire CIDs does open an avenue of attack similar to TCP RST. If we want to have a safe mechanism to request retiring CIDs, then we have two solutions: (1) we require the peer to retire the CID immediately (2) we make this a best-effort request with no enforcement of actually retiring the CID As discussed earlier, (1) is not practical. What does "immediately" mean when it takes time for packets to travel between endpoints? If it means send it on the same packet that carries the ACK, what happens when that packet gets lost? Do I need to send RETIRECID on every single packet in this window until one gets ACKed? I think this quickly because a mess full of corner-cases with security vulnerabilities at play, and I'm not confident we can get to the bottom of it rapidly. Option (2) is similar to STOPSENDING. It's a way of telling the peer that they should do something because what their sending won't do anything useful. NAME I understand why you'd rather have (1) but I don't think the benefits you'll get from it outweigh the security risks. Even if we pick (2), I think everyone will implement this as retiring the CID either in their next outgoing packet or maybe at worst a couple packets later. And that gets you what you want without adding security risks.
NAME I agree with you. My question to you is: what in the current text do you not like? You mentioned before that you'd like me to remove the 3-PTO text. I'm not sure I can do that. If I remove that text, then how quickly is the endpoint requiring the retirement supposed to wait before just retiring the CIDs any ways? If you're saying it's never allowed to retire them anyways, then that's an attack vector by the peer, IMO.
You can start retiring when you receive RETIRECONNECTIONID.
NAME I would phrase the text as:
What if the peer never retires the CIDs? You're saying I'm never allowed to eventually drop the CIDs anyways? The way you have the text, the feature is optional, as long as you're willing to accept a possible performance hit.
I like STOP_SENDING as an analogy overall. NAME I made a suggestion on the PR about what the client should do if it receives a stateless reset token for a retired connection ID. I think that may both allows the sender to immediately retire and provides the peer some advice on what to do if it sees a reset that's associated with an old CID? This approach basically allows the server to wait as long as they think is necessary, and if they really need to stop using the CID, they can send a stateless reset to force it.
NAME That is the catch-22, but retiring them before receiving a retired frame is somewhat pointless. Closing the connection is a security hazard.
I'd say, if you have to wait for the retire frame to remain operational. If you give up, you should use a reset not a close to avoid the security risk.
NAME I think we're all agreed now that explicitly closing the connection is not the answer, because of the security concerns with a delayed packet. But a stateless reset doesn't have those security concerns. Receiving a stateless reset sent in response to a packet using the retired CID only effects the peer if they haven't actually retired the CID yet. NAME STOPSENDING isn't quite the same. When I send a STOPSENDING frame, I've already made the decision to start dropping everything else I receive for that stream on the floor. So by the time the peer receives the frame, a whole flight of data might have been dropped. And we're all OK with that, because it doesn't effect the overall connection state. That's definitely not the case with connection IDs. If I were to start dropping all packets with those retired CIDs first, then best case I lose an entire flight of packets. Much worse if my packet was lost as well. In my scenario, I need to dump these CIDs right now, but the peer is still using them, so I need to give them some time to make the change. I keep the CIDs in my lookup/routing tables and send the frame out telling them to retire. Eventually they acknowledge the frame, so I know they got the new CIDs and the requirement to retire the old ones. This is where we seem to start having differing views. I cannot allow a peer to go on forever using the CIDs I told them to retire. Continued usage of those CIDs is causing all kinds of performance issues, with cross processor locking/contention, for all connections on the effected processors. Since this scenario should be infrequent, I'm OK to accept the perf hit for a while, but eventually enough is enough. I want the text to be clear that if a peer doesn't switch CIDs, then the connection will go away eventually.
NAME yes, but I suggest that you wait for retire frame unless you run out of patience rather than closing you eyes and counting to three. If you get tired of waiting, you can start to reset, but that may depend on the endpoint more than a specific 3-PTO timeout. Hence it goes both ways: receiver should make an best effort to be timely, and sender should have a grown ups patience.
NAME Here is part of your original description of this issue: If the peer keeps using a CID you want to retire, the only impact is performance. So this is in line with what I proposed: keep this feature a SHOULD that is not enforced. Any attempt at enforcing this will open up security risks.
The more I think about this problem, the more pointless it seems to send a stateless reset, especially one that should explicitly be ignored, and one which the sender already knows will have that effect. It's not like that implementation that receives a stateless reset will say "oh bummer, I forgot, I better hurry up!". No, either it is not implemented, or more time is required. A reset just becomes another way of retransmitting a NEW CID frame with a prior to field. So ultimately I think NAME is right in stating "a Should that is not enforced".
NAME can you explain the security risk of responding to an old CID with a stateless reset, sometime after the NEWCONNECTIONID frame was acknowledged? NAME if the peer continues to use the old CID after acknowledging the NEWCONNECTIONID frame, why would it assume the "sender already knows it will have that effect"? The peer doesn't seem to be doing as expected already, why assume anything else at that point? If the peer is ignoring the stateless reset AND continuing to use the old CID, I don't care if they mess things up for themselves. It has solved the performance problem affecting ALL connections on the related processors.
NAME the issue is that an attacker can close the connection if the stateless reset tokens are the same for different CIDs. Perhaps this wording gets you what you want: This allows you to implement the enforcement you want while keeping the spec written in such a way that most implementations don't accidentally allow this attack by being overly enthusiastic in enforcing this.
NAME looks good to me. I will update the PR. Thanks for the proposed text.
Technically, the SRTs don't all have to have been distinct, there just has to be zero overlap between the Retire Prior To set and the still-valid ones. So you could do as NAME has alluded to earlier, using the same SRT for a period of time, then forcing the retirement of all of those because you changed some input to your generation function, and everything you issue new has moved to a different SRT.
Not Martin, but I think this could as easily be different processes/partitions on the same machine. With UDP, they could all be sending from the same source port, after all. Regardless, I think 3PTO is a totally reasonable period of time to expect them to get their coordination sorted out.
Latest changes are looking great, thanks for driving this NAME Let's merge and we can update any further editorial nits when we next do a pass.

Discussion:
migrates away from a local address SHOULD retire all connection IDs used on that address once it no longer plans to use that address. <ins> An endpoint can request that its peer retire connection IDs by sending a NEW_CONNECTION_ID frame with an increased Retire Prior To field.  Upon receipt, the peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frames in a timely manner.  Failing to do so can cause packets to be delayed, lost, or cause the original endpoint to send a stateless reset in response to a connection ID it can no longer route correctly. An endpoint MAY discard a connection ID for which retirement has been requested once an interval of no less than 3 PTO has elapsed since an acknowledgement is received for the NEW_CONNECTION_ID frame requesting that retirement.  Subsequent incoming packets using that connection ID could elicit a response with the corresponding stateless reset token. </ins> 5.2. Incoming packets are classified on receipt.  Packets can either be

The sequence number assigned to the connection ID by the sender. See issue-cid. <ins> A variable-length integer indicating which connection IDs should be retired.  See retiring-cids. </ins> An 8-bit unsigned integer containing the length of the connection ID.  Values less than 4 and greater than 18 are invalid and MUST be treated as a connection error of type PROTOCOL_VIOLATION.

for different connection IDs, the endpoint MAY treat that receipt as a connection error of type PROTOCOL_VIOLATION. <ins> The Retire Prior To field is a request for the peer to retire all connection IDs with a sequence number less than the specified value. This includes the initial and preferred_address transport parameter connection IDs.  The peer SHOULD retire the corresponding connection IDs and send the corresponding RETIRE_CONNECTION_ID frames in a timely manner. The Retire Prior To field MUST be less than or equal to the Sequence Number field.  Receiving a value greater than the Sequence Number MUST be treated as a connection error of type PROTOCOL_VIOLATION. Once a sender indicates a Retire Prior To value, smaller values sent in subsequent NEW_CONNECTION_ID frames have no effect.  A receiver MUST ignore any Retire Prior To fields that do not increase the largest received Retire Prior To value. </ins> 19.16. An endpoint sends a RETIRE_CONNECTION_ID frame (type=0x19) to