Code change:
This is option 4 from the discussion in . That is, making preferredaddress equivalent to NEWCONNECTION_ID with a sequence number of 1 in every way that matters.
Offshoot from review. Servers behind a load balancer, and the load balancer isn't CID-aware. Each server gives out an IP:port per client in the parameter. If the client moves to the SPA, then future migration is fine -- it will continue to talk to the same server using the preferred address. If the client doesn't move, those future migrations will break. There's currently no way to set to "only if you don't use my preferred address". Similarly, if the primary connection is to an anycast address, if the client fails to move to the preferred address, subsequent migration will likely break. Should we blanket recommend that clients not actively migrate if they ignore / are unsuccessful in using the SPA? Or should there be a way for the server to indicate that?
Good question. This brings up a related question for me about disableactivemigration. Given a server can decide not to give out any additional connection IDs, doesn't that give it a mechanism for disabling active migration? Ironically, given recent decisions on connection CIDs and SPA, if an SPA is supplied, then the client always has a second connection ID it can use on any path, not just the SPA. So my suggestion of not giving out extra CIDs fails for the original path, but not the SPA.
What we’ve said for this so far is that SPA is limited to offering a minimum of a single additional CID that needs to work in both places, the original server and the preferred server. As Ian says, the original server should not issue the client any more CIDs (if you have some that are deployment specific) until after they’ve successfully migrated to the new endpoint, at which point the preferred server can handle that issuance. And as he notes, you still have to deal with the fact that there are two CIDs that the client could use, from any incoming address. It seems as though the full way to solve this is to define potentially disjoint sets of TPs that different servers could set to match their particular requirements (around more than just migration). In that world, if you had an initial server that couldn’t handle migration and a preferred server that did handle migration, you could set different TPs to indicate that. It goes well beyond that though, any other TP might have a different values for the original server and then have those be different with the preferred server. But that does feel an awful lot like something that goes with full server migration. In the limited form that we have now, if you want to be able to push someone off to a different server, there are some limitations that you need to deal with, like being able to set up enough of a shared configuration for the two that the same TPs work with both. In practice, I think this means we end up with the following situation: The client either moves to the requested address or it doesn’t. If it does, then you’re in good shape, in theory you gave it TPs that matched what the server wanted and everybody is happy. If it doesn’t, then you’re unhappy, since in theory you had a reason you asked it to move in the first place. You might choose to close it down more aggressively than you would on the preferred server to save resources, or give it smaller limits for flow control, etc. When you’re in the “client didn’t move” state: If the client then tries to use the original 2nd CID that you gave, from the same IP address/port, then a non-CID aware load balancer should be fine. If the client then tries to use the original 2nd CID that you gave, from a different IP address/port, then a non-CID aware load balancer will not be happy. Of course, the problematic case is even narrower, since if the client is probing the new path they’ll see it as though there isn’t any connectivity, just like they will if there’s anything else on the new path that prevents their connection from working — this is always a risk when migrating, so they should be unsurprised and stick with their current path. If they’re forced to move for whatever reason, then it will be just as though they didn’t have connectivity over the new path (which they don’t) and whether or not you disable migration you’ll see the same outcome — the connection is closed. Interestingly, I’m not actually seeing a difference from the client’s perspective in terms of outcome whether you let them try with that CID and fail or if you told them to not try.
In very much shorter terms: isn’t this a situation that all clients already need to be able to handle for other reasons?
NAME I do not like the first option, because IMO banning migration on the original address is going to be a premature optimization based on what we have now, has negative effect on what we would have in the future. My assumption is that the majority of the server deployments would be CID-aware, especially in the long term, especially in terms of traffic volume. Note that the volume of traffic is more important than the number of server deployments, as migration is about (improving) end-user experience. IMO there are two ways forward: (a) State that disableactivemigration TP only applies to the original address. Per-definition, server-preferred address is an address that is capable of receiving packets from any address. There is no need to prohibit clients doing migration when connected to SPA. (b) Do nothing. Clients are RECOMMENDED to implement SPA, and this issue is about a small fraction of deployments that rely on non-CID-aware load balancers. We can ignore the problem.
IIRC, we had that conversation when deciding whether to add in the first place. The short version is that taking this approach prevents the client from rotating CIDs on the primary path; if we want the client to be able to rotate CIDs but not migrate, they need to be distinct signals. NAME in a way you're correct that clients will have to deal with the fallout regardless of what the server says. However, the actual use of is likely to trigger a little more than just "don't do that" -- a client might PING more actively to prevent NAT rebindings, for example. I think it's this hinting aspect that's actually more valuable, since as you note, clients need to be prepared for attempted migrations to fail even with servers that tacitly support them. I don't think this is necessarily true, though I agree that we're talking about smaller and smaller fractions of server space.
NAME While I agree that it might not necessarily be true at the moment, I might argue that it is going to be true. The basis of the resolution for and (both on consensus call) is that servers using preferred address would not be using the server's address for distinguishing between connections. We also know that a server cannot use client's address (unless there is out-of-band knowledge), because clients are not required to use the same 2-tuple when migrating to a different address (and also because there are NATs). The outcome is that the server has to rely on CID when using SPA. And therefore I might insist that servers using SPA would always be capable of handling further migration.
NAME So your suggestion is that disableactivemigration applies only to the original address and the SPA must always support active migration, because it has to use CID routing? I don't think that'd be a problem for our use cases, but I'm not sure about others.
NAME Exactly. I think that that would be the correct thing to do, if we are to make changes. I agree that existing clients might be applying after migrating to SPA, and that they do not want change. But they can remain as they are, because the change in spec does not lead to interoperability issues. To paraphrase, I think making the aforementioned change is strictly better than doing nothing, because it would be an improvement to some, while not hurting anybody.
A truth table might help here. ... --- no preferred address preferred address Right now, our definition of disableactivemigration is a simple "don't migrate". But the argument here says that the server that provides a preferred address is probably capable of handling migrations on the basis that it has to support one. We might insist on the source address not change, but the fact is that some NATs (those with address-dependent mappings) will change the source address. Changing to a definition where disableactivemigration is instead "don't move to a new address unless you use a preferred address" is a little more complicated, but doable. Is it worth the extra complexity this late in the process? Is that complexity justified when it would be trivially possible to define another extension that said "I support migration on my preferred address"?
I'd like to minimize changes, but I believe the status quo is that disableactivemigration applies to both the original address and SPA. As such, all suggestions above seem like a change. There is a mechanism to disable migration on the SPA, which is don't give out any more CIDs once they migrate. I don't expect any hosts to support migration on the original path and not the SPA, so I don't think that's a use case worth supporting(or encouraging).
My proposal is that we defer the ability to signal "I support migration, but only on my preferred address" to an extension.
NAME And leave the current behavior of disableactivemigration applies to both the original address and SPA? I'm a bit concerned that makes SPA somewhat useless. Is this issue a candidate for parking and we can come back to and see if we can live with the status quo?
Which, again, not only disables migration, it disables the ability to rotate CIDs. That's not something we want servers to be doing. NAME argument is convincing that it's improbable for something which supports SPA to not be able to uniquely identify connections (whether by CID or because the SPA endpoint is unique). Therefore, I'd support scoping disableactivemigration to the handshake address, and defer the ability to disable it on the SPA to an extension. Loosening this definition doesn't make any current clients break, so I suspect it's bearable despite the schedule.
If we're going to make a change, I support NAME and NAME direction of scoping disableactivemigration to the handshake address. And I agree that I don't expect this breaks many existing clients.
I personally find the proposed change confusing. I'm not sure I find the use-case of "my original address doesn't support migration but my preferred address does" compelling. Since this can be easily solved by an extension with no downside over it being in the core docs, I don't see a reason to change the spec at this point in the process.
As this has sat here a while, I'll reiterate my proposal again: we defer the ability to signal "I support migration, but only on my preferred address" to an extension. Which means that disableactivemigration applies to both the original address and SPA, so answer the question (sorry Ian, I missed that). I realize that you could change the meaning of disableactivemigration to apply to the address the server uses in the handshake, but there are interesting corner cases and I don't think we need to concern ourselves with those. For instance, if you connect over V4 and get disableactivemigration and serverpreferredaddress. If that SPA includes a V6 address you can't use, along with the same V4 address, what then? Does the server really support migration? I know that the current situation is unappealing, and I think that disableactivemigration is not good for the protocol. But it exists for very good reasons. Trying to limit the scope is admirable, but as we haven't seen a concrete proposal, I think we need to move on.
NAME If this example is concerning, I'd argue that is an issue that already exists, orthogonal to what we are discussing here. I am still not convinced that limiting the scope of disableactivemigration to just the original address is going to cause any new issue. That said, if nobody needs this change, I'm more than happy to move on with status-quo.
I've thought about this, and I don't believe we need this change, but I also think the change NAME suggested makes the current behavior better overall. A server that supports migration on the SPA(ie: unicast) and not on the handshake address(ie: anycast) will end up not disabling migration and hoping that most clients migrate to the SPA or don't support migration of any sort. That seems like a plausible hope, though I'm a bit leery of hope based protocol design.
Thinking about this more, a statement like: "Migration is never guaranteed to succeed. For servers that advertise a preferred address, it's expected migration is more likely to be successful when using the preferred address than the handshake address."
tl;dr: I'm arguing for "do nothing". This conversation has made it clear to me that an address, and not a server, is the thing that supports client migration. Which makes sense -- migration support at the server is entirely a question of routability, and it makes sense that it would be tied to the address in question and not the endpoint. So if we were to do anything, I think I would propose: scoping disableactivemigration to the handshake address, and adding two disableactivemigration flags to the preferredaddress TP, for the two addresses in the TP. That said, this is a clean enough extension. A new extension can define a new TP that provides this granular interpretation of the three server addresses. A server that wishes to support this can send both the SPA and this newSPA TPs, with the SPA TP (with the current scoping of disableactivemigration applying to all server addresses) providing cover for legacy clients. Also I expect that clients are likely to switch to using the SPA if provided. A server that has different routability properties for the two addresses can wait for some deployment and make decisions on whether to risk allowing migration based on data of what it sees as client behavior with SPA. I would argue for doing nothing now. Experience will teach us the right way forward. If necessary, this extension will be easy enough to do, and importantly, the current interpretation of disableactive_migration gives us the right kind of cover in the meantime.
I would rather not make the change in . It assumes that migration is more likely to succeed when attempted to the preferred address as opposed to the handshake address. While this assumption might hold in some server deployments, it is not a fact. I'd rather we didn't bake this assumption into the core spec. I'd prefer to see this feature as an extension.
NAME Would you mind elaborating why a server, accepting connections on a preferred address, might not be able to disambiguate connections using CIDs? As stated previously, when a client migrates to the preferred address, the client's address port tuple can change due to the existence of NATs. A server using preferred address need to take that into consideration. That means that a server has to disambiguate connections based on the CID, which in turn means that the server would be possible to support migration.
Hmmm thanks for that note NAME Looking at things under that light, I agree that the assumption I described above is already part of the protocol. Please disregard my comment above URL .
We don't prohibit the use of a zero-length connection ID in the preferredaddress transport parameter. So a server can use a connection ID during connection establishment, then remove privacy protections by moving to a unique address (see ). Of course, this is largely moot, as the privacy protection offered through the use of connection IDs toward a unique address is nil. In either case, we should clarify whether preferredaddress is exactly like NEWCONNECTIONID, or whether it has its own rules. Right now, it looks like it has its own rules and that is likely bad.
Zero-length connection IDs don't have a stateless reset token associated with them, but the preferred_address TP requires you to set a token. That's an awkward corner case.
That is just one of the strange things, yeah. We should clarify that.
So I see a number of ways forward here: Do nothing. Leave this ambiguous and potentially special. Make the fields in preferredaddress exactly equivalent to a NEWCONNECTIONID with a sequence number of 1. But allow the connection ID to be zero-length. A zero-length connection ID provides a new stateless reset token that is bound to the new address. (This is a weird option that would go against previous decisions to eliminate special handling for this connection ID, but I include it for the sake of completeness.) As 2, except that the new stateless reset token has connection-wide scope such that either of the provided tokens can be used. This is more consistent with established principles. Prohibit the use of zero-length connection IDs in preferredaddress. That is, you can't use preferredaddress without a connection ID. That is, preferredaddress effectively is a new server address plus a NEWCONNECTIONID frame. Something else. From my perspective, I rank these as 4 > 3 >> 1 > 2. But that is largely based on my bias against zero-length connection IDs (as much as it seems like that might be good for privacy, I think that privacy is better served by the use of connection IDs). If people intend to use preferred_address to migrate to a connection-specific address, then I'm OK with 3; we already have to cover the privacy hole that arises from that.
My preference goes to option 4 too. I do not think servers would use both 0-byte CID and preferredaddress TP at the same time. The only case a server using 0-byte CID can also use preferredaddress is: when the server is never accepting more than one connection at a time, or when the server knows that the client's address tuple does not change when the client migrates to a new address IMO that's a niche. And when these cases can be met, I do not see why the client cannot know the preferred address of the server beforehand.
I forgot to add: There is (maybe) a case for using a connection ID during setup and a zero-length connection ID after migrating to a preferred address. That isn't entirely crazy, but I personally would be comfortable losing that option. But I don't operate a server, so I want to give those who do a chance to let me know that this is preemptive.
NAME My personal take would be that people would not do that. When a client migrates to the preferred address, there's fair chance that the client's address tuple will change. That means that the server should be capable of receiving short header packets with 0-byte CID from any address, and determine that the packet belongs to the connection that is migrating to the preferred address. That's possible theoretically, but in practice ... I agree that a server might want to switch to zero-byte CID after migration to SPA completes. But to do that, we need a NEWCONNECTIONID that can carry a zero-byte CID, which is forbidden at the moment. I do not think we'd want to change the spec now to allow that.
That's an excellent point, and a good reason not to allow changing to zero-length connection IDs in combination with preferred_address. The nasty part of that is that in most cases that design would work because clients won't use the "new" connection ID until after the handshake is complete. Of course, it would fail spectacularly if a client did decide to switch earlier - and the client might attract the blame, where it is clearly the fault of the server.
I'll note that prohibiting a 0 length CID doesn't actually ensure the server's IP isn't unique, though it does remove an incentive to make it unique. I'd like to leave the option to switch to a zero-length CID later in the connection, and I think I'd be fine with 4 if we allowed that. An easy example is a large upload. I mostly don't care about the CID, but if I'm uploading a 10GB file, maybe saving some bytes matters. Also, if you're constantly filling the pipe, NAT rebinds are extremely unlikely. Another example is an environment like a datacenter where there are no NATs and routes are very stable, unless a peer intentionally changes it, so the 5-tuple is sufficient. In that case, the ideal operating condition is to typically use a 0 byte CID and switch to a non-0 length anytime a peer changes IP or port. Changing port has some clear benefits in some cases, so it's in no way a threoretical use case. NAME may have thoughts on this range of use case as well.
In our implementation, we plan to do all routing based purely on the CID. IMO, the couple of byte overhead in a 1500 MTU (at least; in datacenter) packet isn't going to make much difference, so it's just simpler to have a common CID routing scheme. That being said, I personally am not against allowing for others to do something different. I've never understood why we make any restrictions on changing between 0-length and non-0-length CIDs at all. IMO, the complexity is that it can change length at all, not that it's zero or not.
NAME we already prohibit changing from non-zero-length to zero-length using NEWCONNECTIONID. I don't want to relitigate that. I want to concentrate on this specific issue.
If we're not going to re-litigate that, then I'd suggest we continue allowing a 0-length CID for the preferred address case.
NAME While I would not argue strongly against continue allowing that, I would appreciate it if you could clarify why allowing a zero-byte CID would be useful. As pointed out in URL, when you provide a zero-byte CID in SPA, you need to be prepared for receiving a short header packet using zero-length CID from any IP address, and be capable of associating that to an existing connection. This is because the client (or the network) might change the client's address tuple when it sends packets to the preferred address. I would anticipate that you wouldn't want to do that in your file upload case (or in any deployment that might accept multiple connections concurrently).
NAME If I know 5-tuples are stable, then using a zero-length CID is always fine, just like it's typically fine in the server to client direction. But in that case, hopefully I can avoid needing to use SPA, making the decision on this issue moot. After thinking about this more, we're probably better off with option 4 for the sake of consistency and if we want to allow 0-length CIDs in more cases, I can write an extension for how that would work, since I don't think it's only a matter of relaxing some constraints.
Thanks all. I think that we reached the same conclusion in the end, via different paths. The proposed fix here is firmly design chairs. A bugfix only, but still worth careful review.
Apologies for getting to this late, but I'm not sure I agree that we should disallow this. Here's a use-case that I think makes sense: the client wants to upload a very large file to the server over HTTP/3 and it has a small MTU, making it particularly sensitive to per-packet overheads. Let's also assume that we are in the future and the server supports IPv6 and has its own /64 (shocking!). In this scenario, a good way to reduce per-packet overhead is to operate the server on a given address, and then switch each client to their own unique server IP using preferred address, and then use zero-length CIDs. I've always found this use-case a good use of preferred address, and I would rather we didn't ban it without a good reason (and making the design cleaner doesn't necessarily count as a good reason at this stage in the process).
That is an example we discussed. As Kazuho noted, that doesn't work as we have already agreed that any connection ID (in this case, a zero-length one) can be used prior to handshake completion. That's why we ended up on 4. As Ian noted, this isn't ideal, but this is an area that is ripe for extensions that might allow this sort of thing.
I don't understand what you meant by the following: Can you clarify? I was under the impression that the client switches to the preferred address after the handshake is done.
Yes, you can't migrate (act on the address information) before the handshake is confirmed. However, the connection ID is not special; it can be treated just like any other connection ID you learn, so you can act on the connection ID information before you confirm the handshake. So that means that a server that opts to include a zero-length connection ID in the preferred_address might receive a packet with the zero-length connection ID at the old address. That might not work out.
What's the point of including a connection ID (and stateless reset token) in the transport parameter if that CID isn't tied to the preferred address in any way? At that point might as well remove that CID entirely and rely on NEWCONNECTIONID frames. My understanding was that the reason for there being a CID in the TP was specifically to make that CID special. Would it make sense to just say that this CID MUST NOT be used when sending to the handshake server IP?
the reason for having a connection ID in the transport parameter is only so that the client has a connection ID to use for the new path
NAME I agree with your intuition, but I think we discussed this in Zurich and ended up concluding the preferredaddress CIDs weren't special. This was in issue and also came up in (which I realize hasn't gone out for a consensus call yet, that PR seems stalled?). In the upload example, the server needs to know that the client is going to send a lot of data, which it might know via SNI, or it might not. I tend to think the more annoying restriction is that you can't change from a non-0 length CID to a 0-length one and vice versa. By supplying a 0-length CID later in the connection along with other CIDs, the client could choose whether it wanted to use a 0-length CID at any time. Thinking out loud, an easier solution may be a TP like 'acceptemptyconnectionid' that indicates at anytime, the peer can switch to a 0-length CID if it wants. It could either think the 5-tuple is stable or its happy re-establishing the path with a non 0-length CID after a NAT rebind. In this world, the Connection ID is more of a path re-establishment token.
Thanks for the details NAME and NAME I get a sense I might be trying to re-litigate past consensus without providing new information, so I'll back down. I do think that the ghost of hypothetical linkability has caused us to make some very odd design choices, but this isn't new to this issue/PR. In the end, this is the kind of feature that can be enabled by extensions so I can live with us further restricting the base spec for consistency.

Related discussion:
text: connection ID when the client initiates migration to the preferred address. <ins> The Connection ID and Stateless Reset Token fields of a preferred address are identical in syntax and semantics to the corresponding fields of a NEW_CONNECTION_ID frame (frame-new-connection-id).  A server that chooses a zero-length connection ID MUST NOT provide a preferred address.  Similarly, a server MUST NOT include a zero- length connection ID in this transport parameter.  A client MUST treat violation of these requirements as a connection error of type TRANSPORT_PARAMETER_ERROR. </ins> The active connection ID limit is an integer value specifying the maximum number of connection IDs from the peer that an endpoint is willing to store.  This value includes the connection ID received