Original text:
when it receives a packet with a previously unused connection ID.  An endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of <del> connection IDs; see reset-token. </del>  An endpoint that initiates migration and requires non-zero-length connection IDs SHOULD ensure that the pool of connection IDs

Reviewer comments:
Add another note when cautioning endpoints about how many CIDs to issue: in this case to note that if you issue a CID you need to be willing for the other endpoint to use 100% of the issued CIDs, which comes with an implicit willingness to maintain state for each path on which that CID could be used. (That's one per path, but # paths == # CIDs in the worst case.)
This adds a MAY and the issue is already marked design, so I'm inclined to mark this design and do a consensus call on the issue with the next batch.
We could also leave this as choose, which seems completely reasonable as well. The only reason I went for was because the previous sentence is the same level of advisory and we also used there.
NAME I'd prefer you keep it as is. The process isn't going to cause an appreciable delay in this case.
Update: I thought simultaneous path validation wasn't necessary, but it looks like I misunderstood. Changing this to an editorial issue so I can re-read this text and hopefully make it clearer. I believe this is already implied, but I couldn't find it stated anywhere. I think it'd be ideal to set expectations that servers are not expected to validate multiple paths at the same time. Validating multiple paths seems like a plausible feature for multipath, but adds undue complexity for connection migration.
Does that imply that a client can only probe a single path at a time? What's the time it has to wait after apparent unsuccessful probing of one path until it can probe the next?
NAME I'm not sure if I agree. It has been my assumption that a server is expected to respond to clients probing multiple paths concurrently. Note that in , we added the following text:
Hmm, I didn't think this was already implied. You're not expected (or allowed) to send non-probing packets on multiple paths, but you can certainly get probes in from multiple different remote endpoints. This is somewhat necessary if we're to be resilient to the various packet racing attacks -- if an attacker can cause you to be upset about a packet coming in from a different destination then they can really mess you up.
I'm a bit confused about why a client needs to probe multiple paths simultaneously, given we have connection migration, not multipath. To me, it just adds complexity. I guess I can just give out a single CID to mitigate the issue, but do we really expect servers to correctly implement probing on multiple paths? What if they all succeed? A client can still only use one. NAME can you clarify the packet racing issue a bit more or point me to text that already does? I'd like to draw a state transition diagram for path validation, but if we can have multiple paths being validated at once, I'm not sure that's practical.
NAME So the scenario is that a client is using one path and has 2 or more new ones available? I've heard future hardware could make that possible, though I've never seen that case myself. The short of this is now that we're trying to move to IETF QUIC style connection migration and path validation, and I'm finding it fairly confusing.
I think there's a pretty big distinction between probing a path to establish the potential for connectivity and migrating the connection to that path. For example, I would not be at all surprised if a client were to probe all available paths and choose the "best" one to migrate to when the need arises (often, although not always, there is some warning that conditions are worsening on the current path which provides the opportunity to evaluate alternatives). That said, I think a large part of the reason to allow probing on multiple paths is to avoid requiring synchronization of a shared understanding as to the state of current probes. We don't want to add a bunch of machinery to establish failure of a probe on a different path. So even if we don't think that a client would ever intentionally probe multiple paths, if a client probes one path and hears no response, it seems completely reasonable to think that it would switch and probe a different path if it had one available. If it turns out that there was just some delay for that initial path, say it had a 1.5s RTT, now the server is going to think the client probed two paths at the same time, when in reality the client thinks it's just talking to one. It also doesn't seem particularly harmful to allow this? As NAME noted, there's some good text in there about limiting state required on the server, and of course path validation can happen at any time, so it seems completely reasonable to "forget" that any given path was validated if a client probes and then doesn't end up using the path. (Note that you'd likely want to retire the CID used on that path, so as to replace it with a new one when you did that forgetting, which also serves as a nice heuristic for the other endpoint to say "oh, hmm, they don't think they're talking to me over there", but we can dig into that later.) The packet racing thing is a reference to and the previous section. This led us to require endpoints to validate both the current and the new path when it changes, which conveniently provokes some traffic to reconfirm the active path and makes it more difficult to disrupt a connection that's gone idle. While technically this is only discussing one "new" path and then the old path being validated at the same time, it seems as though it would be trivial for such an attacker to convince an endpoint that it was seeing "probes" (since we've discussed that it's not that difficult to identify a probing packet as an observer) from multiple paths and therefore be quite upset if such behavior were restricted. I'm not saying this happens all the time, but it seems pretty reasonable to have a Wi-Fi interface and then a dual SIM device with multiple cellular interfaces. Overall, it seems as though (even if this isn't crazy common) it's not unreasonable to expect that this behavior might happen, and it doesn't seem like there's a big benefit to restricting it, nor harm in allowing it, beyond the existing text reminding endpoints to be careful with the amount of state they're willing to allocate to any individual client (which is a thing outside of any probing/migration concept anyways).
+1 to what NAME says. The resolution of (which ended up as PR ) was that a server has to remember state of N paths where N is the number of CIDs active at one time. As NAME says, a client cannot tell when a server thinks that it received two probes concurrently. Therefore, it is very tricky to have this kind of restriction within a specification that allows clients to retain multiple backup paths.
I agree with NAME and NAME here. If a client sends a probe on interface A, and then interface A disconnects and then interface B appears immediately, the client should be able to probe on interface B even though to the server will receive two independent probes at roughly the same time. Servers need to be able to handle multiple concurrent probes in order for migration to be reliable.
Note that while the server needs to respond to the probes (so the client can validate the paths), I don't think there's any requirement that the server continue to regard all those paths as validated. When the client actually migrates, if the server wants to treat the migration as an unvalidated path and initiate validation again, that seems fine.
Well, the paths aren't validated until the server explicitly validates them, so it seems like the cost is pretty minimal until the server decides to engage resources for that path. The server can also respond on the old path if the client is just probing.
NAME do you want to think about how you reached your conclusion here and see if there is something we can do to avoid that? It seems to me that the cost to a server is limited by how many paths it wants to probe, not how many the client probes, so the state is well managed. It's a little less neat if the client actively migrates multiple times. But I would say that the server can abandon probes to keep the number of active probes down to at most 2 (one for the most recent validated path, one for the most recent active path). Is that something we need to explain more thoroughly?
I have read this text quite a bit, but not end-to-end in quite a long time, so I believe that's the cause of my confusion. I'm planning to re-read all the path validation and connection migration text and I'll send out a PR if there are points that I think I are unclear.
NAME have you had a chance to do this?
Unfortunately, not yet.
Note that man-on-the-side attacker can take a valid packet and resend it from a different address. The crypto does not protect against that. If the attacker somehow isolates probe packets, the server will end up receiving probes from a variety of endpoints. Better not do something silly in that condition.
Would people please take a final look at the PR?
At the moment, it is my understanding that we do not give any advice in particular. But to me it seems that there would be issues if the number of probed paths that the server remembers is less than the number of spare connection IDs it issues. Consider the case of a server that advertises three spare connection IDs, but retains information of one probing path (i.e. remember one set of 5-tuple that has been probed, and the connection ID being associated to that tuple). If a client connecting to this server probes three paths concurrently, and if it keeps on sending PATHCHALLENGE packets periodically on each of those paths (in order to keep the holes in NAT open), two issues arise: The connection ID pool of the server would run out quickly, as the server would use a new connection ID for each probing packet it sends in response (because each probing packet sent by the client would have a different source address than the previous packet). This is an issue, as we state that an endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of connection IDs_ (). The purpose of path probing is to prepare an alternate path beforehand. That benefit diminishes if the server remembers only one (or few) of the paths being probed. Instead of recommending servers to remember paths as much as the number of spare connection IDs that they issue, we can introduce a transport parameter that caps the number paths that a client may probe. But IMO having a new TP seems unnecessarily complex, because I think that servers can simply associate path information to each active connection ID it maintains.
Given QUIC v1 is chartered to support connection migration and not multipath, we could also say clients shouldn't probe more than one extra path at once. I do not want to add another TP for this use case, because I don't think it's sufficiently valuable.
I'm fine with having a hard-coded limitation, though if we are to set the maximum to one, then we need to decide whether or not migration due to SPA is counted. It would be simpler to include it, but the potential downside would be that probing using a different network interface would be forbidden until migration to the preferred address concludes.
So a server doesn't need to probe in response to a client that probes a path, so it doesn't really have any state to attribute to probed paths. A server can just respond to probes. The cost to it, if it doesn't, is that it has to probe after a migration and that could have performance implications. I would be happy saying that an endpoint ideally both probes and remembers the status of those probes for as many paths as it has issued, active connection IDs. Probing paths and remembering the status of probes partly mitigates the reduction in CWND that follows migration, so there is a potential performance gain from keeping this state. But the window is still likely reset, so the performance hit is essentially unavoidable either way. The total number of paths a server might have to remember is limited by the activeconnectionid_limit of the client, but it can be less (as you already noted), so the server has the ability to limit its exposure. Thus, a server can probe and remember the status of as many paths as it likes. I'd say that this is a fine piece of advice, but that's all it needs to be. I agree that yet another transport parameter is not a great reaction to this relatively minor issue.
NAME I think I disagree with the observation. The problem of responding to a probe without retaining state for that path (in addition to this performance implication) is that the server burns a CID for each probing packet. As stated in the , the worst case scenario is that CIDs will be consumed at a high rate when a client probes multiple paths simultaneously.
Personally, I don't think this worst case scenario is very bad at all. But I'd support adding some cautionary/explanatory text.
I don't think migration vs. multipath is currently the distinction for how many paths a client probes, it's the restriction on how many paths the client uses at any one time, and I think that's appropriate. There are many cases where I'd want to be able to probe more than one path, and assuming that a device has a single network attachment (where we need to include IPv4 and IPv6 as more than one given that you need a new CID for each local address) doesn't make sense either. The number of paths that a client can probe simultaneously is limited by how many CIDs the server is willing to issue to that client. To the point of the original issue: If adding that TP is unnecessarily complex (and I agree that it probably is), what's wrong with recommending servers to remember paths up to as many as the number of spare CIDs that they issue? The whole point of issuing more than one spare CID is that the other endpoint may need to probe additional paths or those probes may be lost, but if you aren't willing to deal with things coming in from 100 places, don't issue 100 CIDs. Obviously that's an exaggeration, but 2/4/6/8 for each client puts a nice bound and is roughly what I'd expect to be sending in a TP anyways. There's already a mechanism where the person doing the probing can "retire" that CID and signal to you that you no longer need to have that state around, and that's tied directly into their ability to generate new state that you'd need to keep around. In essence, I think I agree with NAME statement above, if a server needs to reduce its exposure further it can do so, at the potential cost of performance or burning CIDs faster (as NAME points out). Enhancing the text around the existing sentence in Section 5.1.1 (that cautions against issuing too many CIDs) to also say that a server needs to maintain this state and should consider that when issuing new CIDs could be reasonable, but I wouldn't consider that time for a new TP?
I tend to agree with Eric, but I think there are some aspects to consider: You don't want to limit yourself to one extra path as Ian suggests because you might have 3 or 4 paths forward, e.g. sitting on WLAN 2.5GHz with optional 5GHz if the first becomes congested, along with a 4G and 5G modem path. You need to monitor these and choose the best path forward once the preferred path fails. Maybe this is a bit extreme, but just to illustrate path migration isn't linear. At the same time, the peer needs to issue enough IDs to not stall progress in addition to accommodating the number of paths forward a typical client might need. I'd say having double the number of CIDs in spare is probably OK, but maybe there isn't a need for a spare CID for every path. So it could be number of paths + 2 or something. Thus, a recommendation could be that a client SHOULD not track paths beying having 2 spare CIDs and a server SHOULD be prepared to track the number of issued CIDs minus two, as a general rule. Exceeding that may require delayed probing which could lead to temporary delays.
BTW: a related migration issue: a client might have to track separate root certificates for separate paths because one path could be on a corporate WLAN with substituted root certificates for traffic monitoring while the other path happens over 4G or 5G. This is probably not something that can be supported on a single connection neither in QUIC v1 or v2, but it is actually a concern in a real use case I'm aware of.
I tend to agree with what NAME says. If I haven't been clear, I share the view that probably the best solution to this issue is to recommend servers to remember at most N paths, where N is the maximum number of active connection IDs it provides to a peer at any given moment. I think NAME is correct in pointing out that the client might not use all the spare CIDs for probing, but I would argue that that can be a local decision of each client implementation. In terms of operational complexity on the server-side, the recommendation being N or N - 1 or N / 2 does not matter at all. Let's simply recommend N for the servers, and be silent about what clients should do.
I think it makes sense to simply advise that servers should remember as many paths as the additional CIDs they hand out, as NAME suggests. NAME can you write up a PR for this?
Sure, will do! I'm thinking this will live right next to the issuing CIDs bit: as we note that you may want to rate limit, we can say it would be a good idea to recognize the state that may be associated with each CID, in this case, the path validation status (which is needed for the validating paths section anyways, since we say you might have remembered it from previously) and any other per-path state that you might keep.

Revised text:
when it receives a packet with a previously unused connection ID.  An endpoint MAY limit the frequency or the total number of connection IDs issued for each connection to avoid the risk of running out of  <ins> connection IDs; see reset-token.  An endpoint MAY also limit the issuance of connection IDs to reduce the amount of per-path state it maintains, such as path validation status, as its peer might interact with it over as many paths as there are issued connection IDs. </ins> An endpoint that initiates migration and requires non-zero-length connection IDs SHOULD ensure that the pool of connection IDs