Code change:
Note also the editorial changes in , which are great and still needed: this adds a little bit more to the reference from those changes to do the key clarifying for . The remaining clarification about is coming in the SPA PRs, but I wanted to look at this paragraph in one place.
Section 12.2 makes two quasi-conflicting statements about coalesced packets. In a connection where the sender has multiple CIDs in hand, using two different CIDs in the two coalesced packets is entirely consistent with the guidance to the sender, but would be dropped based on the guidance to the receiver. We should align these two statements.
Fixing this would require a change to normative language. I'm of the opinion that the first should change to different connection IDs rather than connections; the second might change to a MUST NOT, but it could remain a SHOULD and achieve the same end. The main reason for this is that routing will be based on the first packet. But there is also a secondary benefit in not creating a strong correlation between different connection IDs for the same connection.
But this complicates the coalescing logic, because you no longer only need to look at packet types but now also CID fields. At least in my stack, that is an ugly change to make at this stage.
The alternative, which I can now see is probably a smaller change is:
What motivated this was two observations: we don't coalesce except during the handshake, where we are not migrating some stacks generate packets for coalescing while processing, and so they might generate a datagram with multiple connection IDs
NAME writes: This is much simpler alternative than what is being proposed. Now an implementation can only drop a packet after doing a connection lookup, whereas before it could just compare the CID to the previous value. If this is just a matter of personal preference, I vote for this approach, so now the score is 1:1 (with NAME being on the opposite side). To take a step back: why is recommendation to drop packets exist in the first place? If a sender wants to coalesce packets from different connections, why not let it?
I think it should not allow having multiple packets for different connections in a single UDP datagram - please. Currently I see implementations (including mine) have something like: Receive UDP packet. Parse packet header. Check CID, find the connection handler. Pass the packet to the respective connection (thread) to handle the packet. I think it makes implementations much more complicated (already). Also I wonder it may cause security issue if having bad assumption?
The classic example would be coalescing Handshake packets, whose DCID is the SCID used by the server, and Initial or Zero-RTT packets, whose DCID is the client-chosen Initial DCID. The client MAY switch to using the server chosen CID for Initial packets. Maybe we should say that it MUST do that if it is going to coalesce Initial and Handshake packets. Not so clear about 0-RTT packets. Is it actually allowed to send them with a different DCID than the Initial DCID?
NAME : Our assumption has been that load balancers, for instance, can simply route on the CID in the first header they see in the UDP datagram. If we allow different connections to be coalesced, that complicates things. It may also start to leak some things about the way load is distributed within the server network: if some CIDs when combined cause loss and some don't, that reveals some things about how the server load is distributed. Is there a strong argument for changing this behavior? This new language in was always the intent. I suspect it didn't change when we started allowing multiple CIDs within a connection. Unless there's an argument for changing behavior, this is really an editorial PR.
NAME If I understand correctly, every time you change the connection ID, you change the connection ID for all packet number spaces. Assuming that my reading of the spec is correct in this regard, it seems like there's no compelling reason for making this change?
NAME the spec actually says (7.2): The first flight of 0-RTT packets use the same Destination Connection ID and Source Connection ID values as the client's first Initial packet. Upon first receiving an Initial or Retry packet from the server, the client uses the Source Connection ID supplied by the server as the Destination Connection ID for subsequent packets, including any 0-RTT packets. SO you are correct, there is no ambiguity.
NAME Thank you for looking this up. Then I'm opposed to making any change here. Not only is it unnecessary (that alone should be enough), it would also make it more difficult to evolve the way we deal with coalesced packets in the future: I can imagine a future QUIC version introducing a way to authenticate the entire coalesced packet, while at the same time compressing the header. This will be easier if we keep the explicit property that all connection IDs of the coalesced packets match.
NAME I'm not sure if I agree. Coalescing of QUIC packets is a feature specific to v1, entities that operate without the knowledge of the connection (e.g., load balancers, datagram dispatchers) are not expected to consult the CID values of QUIC packets that are appended. Therefore, what we are talking here does not affect what we can do in the future.
That argument was based on the thought that it will be easier in a future QUIC version because the diff to v1 would be smaller. My main point is that this change is unnecessary because when endpoints change connection IDs, they change it for all packer types at the same time.
NAME Thank you for the clarification. I think I do not agree to this, too. IIUC, the consensus we reached with was that the CID of a Handshake packet and that of the 1-RTT packet can be different. While it is true that the initial DCID set by the client is expected to change when the client receives the first "active" connection ID supplied by the server, the fact does not mean that the CIDs of all QUIC packets within a datagram has to be the same in all the cases.
I think we need to make a change, in one direction or the other. It's unfortunate to mix CIDs, since it clearly links them to any observer, but doesn't seem like a big deal. For some implementations, it sounds like it's a big deal to avoid mixing them. Therefore, while I'm content with either resolution, it seems like it makes more sense to resolve toward being more permissive; e.g. . Mixing packets from different connections seems like a generally bad idea, since it assumes an implementation structure on the far end for it to work. The more likely outcome is that coalesced packets from other connections will simply be dropped as invalid. (Of course, this very property could be used for padding / misleading observers.)
Hi Mike, Can you say more about why you don't see this as an issue? Is it because this use case is most likely during the handshake? Or is there some other linkability already present that makes this less of an issue? regards, Ted
Some implementations try to change the CID as soon as possible, to avoid linking 1RTT traffic with the CID visible during the handshake and to improve privacy. There will be a brief window when they still need to send both some handshake packets and some 1RTT packets. But coalescing a Handshake packet with the old CID and a 1RTT packet with the new CID links old and new CID and thus negates the effort. And just in case someone asks, no, you don't really need to decrypt the packets to access the second CID. You just need the coalesced packet and another 1RTT packet. Not hard.
NAME If that is the concern, wouldn't it be better for endpoints to migrate to a new CID after learning that the peer has dropped the Handshake keys? If an endpoint switches to the new CID before the handshake keys are dropped by the peer, there's chance that the peer might use a new CID in the Handshake packets that the peer sends. That at least leaks when the connection was established for that CID.
NAME yes, the simplest way to not run into coalescing problems is to not start migrations or CID changes before Handshake Done.
Apologies, I'm getting lost in the weeds here. I agree with NAME that we need to make a change, and that is really meant to fix the text to what the intent always was. NAME NAME NAME Are we all agreeing on this point?
NAME That is my view. I think to be an editorial amendment of . Regarding linkability problem, the simple and safe way of mitigating the problem (as proposed in URL) does not interfere with the proposed resolution.
NAME I personally prefer the old version, force single CID in coalesced packet. Loosening the text to allow multiple CID requires a code change in my implementation.
I'm with NAME Single CID.
Both, actually. This occurs during the handshake, when addresses are required to be stable. Unless you're running a large number of parallel connections, all CIDs used on that 4-tuple during the handshake belong to the same connection with very high probability. You can break that linkage by deliberately changing CID and port at the same time, but you can't do that until the handshake is confirmed. If you want the linkage broken, you need to do that jump post-handshake anyway.
Hi Mike, I think "high probability" is probably app-dependent, because there may be cases where there are significant numbers of parallel connections. In the case of proxied queries, for example, the use of the same 4-tuple but a different CID could turn out to be common, because the proxy may prefer to keep traffic from its different clients separate even if they could be served by the same upstream server. A DNS example: if you have an DNS resolver talking to an upstream proxy or a common authoritative server, that resolver may want to set a different Client Subnet for different query streams, so that input can be used to determine the preferred reply. In the presence of a NAT or CGNAT, there could be interesting interaction with the dwell timers in the state table of the middlebox. If you change it too late, the middlebox may recycle the most recently freed port, which could result in you getting the same 4-tuple to an observer on that part of the path. Speaking strictly as an individual, I think using a single CID per coalesced packet as Christian prefers is cleaner and leaves open app behaviors we may not have yet identified. regards Ted I'm not a big fan of Client Subnet, and these particular parallel connections may or may not pose a risk; it's just a proxy situation where the parallelism might occur.
The discussion is active, I'm marking this as design.
NAME I might be missing something, but if an endpoint wants to cut linkability between the packets used during the handshake and the packets used in 1-RTT, that endpoint has to switch to a new CID after the peer has dropped its Initial and Handshake keys, regardless of the outcome of this issue. That is because if the endpoint switches to a new CID while the peer is still sending Initial or Handshake packets, we allow the peer to use that new CID in the long header packets sent in response. Therefore, I do not think that requiring endpoints to use one DCID in all the coalesced QUIC packets help us in practice.
In my implementation, ngtcp2, it processes packet in the way that PR proposes; which means I just over looked existing text, but anyway, I think that processing a packet atomically is cleaner approach in implementation wise rather than passing a some kind of pointer to the very first packet header.
NAME Once a non-coalesced packet is passed to a connection, in my implementation I don't have to do any checking. The server already makes sure to pass packets to the correct connection based on the DCID. So there's additional validation logic required for coalesced packets. And the easiest way is to just check that all DCIDs match. Sure, it would be possible to somehow query which other DCIDs would also be valid for the same session, but it's more complicated. And given that nobody has come up with any good reason to coalesce packets with different DCIDs in the first place, I'm wondering how this additional complexitiy would be justified.
I don't foresee a large problem with the proposed change, but I also don't think it's necessary.
Do we have a proposed resolution for this one?
I think we settled on requiring connection IDs to be the same, but I haven't had the time to write that up, so not just yet.
NAME FTR, discussion happened on the mailing list . My sense is that it's still not clear to some (including me) why the current fix in (requiring "connections" to be the same) doesn't work, but nobody is arguing strongly against what NAME said above (requiring "connection IDs" to be the same). So that might be the least common denominator.
added the text " Connection IDs that are issued and not retired are considered active; any active connection ID is valid for use at any time, in any packet type. " As far as I can tell there is no text specifying that an issued connection ID cannot be used with subsequent connections. The introduction to connection IDs state that there must not be correlation across connections, but if an issued connection ID was never consumed, there would be no correlation because it was transmitted encrypted.
We're currently not specifying which DCID is supposed to be used on retransmissions of Handshake packets. A client might receive a NEWCONNECTIONID frame in the first couple of 1-RTT packets it receives, before receiving either a HANDSHAKEDONE or an acknowledgement for a 1-RTT packet is sent itself. Therefore, it's still running loss recovery on the Handshake packets it sent out earlier. If there's an acknowledgement outstanding for one of those Handshake packets, the client will have to retransmit that packet. Which DCID is it supposed to use on the retransmission? Is it the DCID that was used on the original packet? Or is it the CID provided in the NEWCONNECTIONID frame? Note that the NEWCONNECTION_ID frame might already have requested the retiring of the CID used during the Handshake.
As I can see, there are three options: a) let the receiver of NCID to send Handshake packets using the new CIDs b) require (or recommend) receiver of NCID to continue using the original DCID until the handshake is confirmed c) require (or recommend) the sender to withhold the emission NCID (with a non-zero RTP) to until it sees an ACK for handshake confirmation I prefer (c) (or maybe (b)), because (a) has corner cases even though it might seem easy. Consider the case where a server sends TP.preferredaddress then as the first packet, sends the NCID frame with RTP set to 2. If we adopt (a), this would mean that the client would have only one CID that it can use (the one being embedded in the NCID frame) even though it needs two (one for the original path, and another for the path specified by TP.preferredaddress). As you can see, the design of the NCID has been based on the assumption that providing one new CID alongside a retirement request guarantees that the receiver would have enough active CIDs. But in this particular case, that is not the case. Based on this observation, and based on my assumption that a non-zero RTP would be sent by only some of the endpoints (the necessity of CID retirement is questionable for HTTP/3, the scope of QUIC v1, as the HTTP connections are typcially short-lived and can most likely be gracefully terminated to induce a new connection when retirement of CIDs is necessary), my preference goes to (c): an endpoint that uses RTP should cover the cost.
NAME (I think you mean RPT, not RTP) Thank you for the analysis. I prefer (c) as well, except that I do not want to condition it on non-zero RPT, to keep things simpler. Otherwise we're still left with what to do at the receiver. Ensuring that the handshake confirmation is acked means that there are no HS packets pending.
I'll also note that this issue was found through a resilience test we've built for the interop runner -- high loss handshake test. We're excited to see how everyone's handshake machinery performs under 50% loss :-)
At the time of this writing, ngtcp2 does (b).
NAME for you problem with (a) isn't that just a general problem with having multiple paths and then getting a NEWCONNECTIONID with RPT for all previous CIDs? You end up in a scenario where you only have 1 CID (the new one) and multiple paths that just had all their CIDs retired. If you can get away with using that one CID for the active path, you can still make forward progress. If not, the connection dies.
I believe we specify (b) already: "A client MUST only change the value it sends in the Destination Connection ID in response to the first packet of each type it receives from the server (Retry or Initial); a server MUST set its value based on the Initial packet. Any additional changes are not permitted; if subsequent packets of those types include a different Source Connection ID, they MUST be discarded. This avoids problems that might arise from stateless processing of multiple Initial packets producing different connection IDs." URL
NAME that is just indicating when a client should change DCID in response to receiving a Initial/Retry packet with a new CID. I don't think anything restricts either side from sending NEWCONNECTIONID frames in their first 1-RTT packets and their peers immediately using them for any packets (including handshake) they send. Personally, I don't have a problem with (a). I think that no matter what receivers will have to handle the multiple paths but single CID scenario, so I would prefer not to special case the handshake. I also don't think we have handshake confirmation problems because an endpoint should only send a NEWCONNECTIONID out when it's able to accept it. Then, if the peer can decrypt the packet with the new CID, I see no reason why it shouldn't be allowed to use it immediately.
NAME my reading of the text I quoted is that you cannot use any DCIDs in long header packets that weren't either the original client DCID/SCID or from a server Initial or Retry.
NAME There's a fourth option: d) require that the sender withholds the NCID frame until it has confirmed the handshake. As soon as an endpoint confirms the handshake, it drops the Handshake keys (I'm assuming here). As a result, the endpoint obviously doesn't care about Handshake packets any longer (and especially doesn't care which DCID was used to send the packet). Therefore, there's no problem if the receiver of the NCID frame immediately retires the connection ID used during the handshake, even if the the NCID and the HANDSHAKE_DONE frame were reordered.
NAME The part about what to do with DCID changes is underspecified. From the part of the spec you quote -- "Any additional changes are not permitted; if subsequent packets of those types include a different Source Connection ID, they MUST be discarded." -- it's not clear what these additional changes are limited to. I think the intent was for all long-form packets, but that's certainly not clear here. In any case, the problem with that approach is that the client will have to carefully construct HS packets with the old DCID while sending everything else with the new DCID.
NAME : (d) works too, but an endpoint still has to decide which DCID to use for sending HS packets in the case that the HANDSHAKE_DONE frame is lost. I'd rather not leave that unspecified.
"of those types" => Retry or Initial, since that's what it was talking about. That's trying to avoid the case where the server statelessly generates a new Retry (or Initial, in the rare case that would be stateless) with a second SCID upon receiving a retransmission or duplication of the client's Initial. But it doesn't say anything about the server issuing new CIDs via an NCID frame in 1-RTT before the handshake is confirmed. I think (a) or (b) would be fine; the server is clearly intending to accept the other CID if it issued it, but could conceivably use separate logic for long-header versus short-header routing.
Can someone explain why using the new connection ID is a problem?
NAME I think that that is a good way at looking into the issue. I agree that the endpoints can start using the new CIDs during handshake, if the peer sends NCID frames. The only problem that I can think of is related to retirement (see URL). When the server provides TP.preferred_address, and also decides to retire the CIDs immediately after sending all the handshake transcript, there's going to be a very small time window in which it has to send two CIDs (one for the current path and another for the path using the preferred address). Unless the QUIC stack allows bundling of two NCID frames when building a QUIC packet, the client might end up not having sufficient amount of CIDs, in which case it might terminate the connection. It is my view that this is very different from the ordinary case of retiring CIDs, in which case there would be enough time window to guarantee that a handful number of new generation CIDs are delivered before the older ones are delivered (cc NAME That said, I now tend to think that the most we might have to do is the following two: clarify that CIDs received using NCID frames can be used for Handshake packets have a cautionary text stating that a server needs to make sure that the client has enough number of new generation CIDs when it retires an older generation of CIDs
NAME Wouldn't we be able to solve this by using my proposed option d?
Artificial constraints on when the frame can be sent would be terrible to enforce and would introduce a performance cost for using NEWCONNECTIONID. In a great many cases, the 0.5RTT from the server is idle, which makes it a great way to get things like NEWCONNECTIONID sent without taking capacity from real work. Asking the server to defer sending would mean that NEWCONNECTIONID could compete with HTTP responses and the like. I see no reason not to allow use of connection IDs when they are available. They apply to the connection as a whole. You will note that we allow changing them with the Initial/Retry mini-protocol in ways that don't correlate with other connection-level events, so this isn't any different in that regard. Forcing the connection ID state to synchronize with other state changes seems more likely to complicate things than help. The caution NAME mentions is always true. Whenever an endpoint is forced to retire connection IDs, a short supply is always a risk to the connection.
I think QUIC allows more than 1 NCID frames in a single QUIC packet. Now my preference is (a). If server utilizes retire-prior-to, it should know the consequences and should provide sufficient backup connection IDs. Otherwise, it would be misconfiguration of server software.
NAME The problem is that the client might end up receiving only some of the NCID frames, unless the server always sends (and retransmits) the set of NCID frames as a whole. But anyways, I agree that (a) is probably fine. That is because it is trivial to design a server that never requests CID retirement during the handshake. All you need to do is start issuing the new generation of CIDs at least N seconds before sending NCID frames that asks for the retirement of the previous generation, where N is your handshake timeout. Knowing that this design is why I initially preferred (c), not realizing that it works equally well with (a). But know that I understand that, I prefer (a) as it is simpler.
Given a server has to sent NCID to have the CID change, I'm fine with (a), but I thought it was not what we'd previously agreed to and specified, so I believed (b) was the status quo.
I don't think this is a serious problem, because if the server doesn't want the long header DCID to change, it can wait before sending NEWCONNECTIONID. That said, we do have an issue here in that different readers of the draft had different interpretations of what was allowed. All that to say I don't much care if this is allowed or prohibited, but we should at least make an editorial change to clarify things.
I agree with NAME here. Any alternative is ok, but the draft should say whether servers can expect that the client might switch DCID and whether the clients can expect the server to send NCID w/ RPT>0 before the handshake is complete.
Let's keep it simple: no migration before Handshake Done, period. That would include prohibit NCID in 0-RTT Also state that implementations MAY drop any packet received with NCID different from original before Handshake Done.
NAME The problem is not that simple. Consider the case where the server receives ClientFinished and sends two 1-RTT packets, the first packet containing HANDSHAKE_DONE, and the second packet containing an NCID with RPT set to 1. If the first 1-RTT packet gets lost, the client receives a request to retire the connection ID before it confirms the handshake.
I think I'd prefer a design that meets the two requirements stated below: A server should not be required to withhold NCID frames until it receives an ACK for HANDSHAKEDONE. A client should be allowed to use the original CID until the handshake is confirmed. The intention is to minimize additional complexity to existing (and future) QUIC stacks. One solution that meets these requirements is to state that a NCID frame confirms the handshake, exactly the same way as HANDSHAKEDONE frame does. That would be a trivial change to any of the QUIC stacks, and there would be no ambiguity regarding the state changes.
Discussed in ZRH. Proposed resolution is to clarify that issuing a CID means being prepared to seeing it used at any time. Additional Editorial issue on packetization issues.
Editorial PR on packetization challenges with retransmitted frames is .
Ugh, editorial changes for a design issue. Reopened for consensus calling.
This was fixed by . is related to this issue, but that simply clarifies things. To keep things sane, let's just close this when is merged.
Looks good. URL discovered a minor "discrepancy" in our requirements for coalescing packets. We required that: senders not coalesce packets for different connections in the same datagram. receivers reject packets with different connection IDs in the same datagram. URL was the result of a discussion where I was convinced that looser constraints here weren't harmful. The requirements would identify connections, not connection IDs. Originally, my inclination was to fix the requirement on the sender to use connection IDs, which would remove some linkability. But then I was reminded that linkability is already defeated for several reasons. You can't coalesce outside of the handshake and we require a stable path for the handshake AND long headers include a source connection ID that can be used for linkability. So the linkability concern is basically lost. We also learned that some implementations were generating packets while processing incoming packets. Later they would coalesce these into a datagram that might then have different connection IDs on the packets it contains. There has been some opposition to the proposed resolution in PR 3870. Apparently, for some, having multiple connection IDs in the same datagram complicates processing. I don't understand this objection. It seems to me more difficult to retain state across packets than it is to process each atomically. I was hoping that Christian or Nick can explain more about how this affects them. Marten also indicated that there was a reason not to allow this, but if I understand that, this is based on some potential for future optimization, which could be conditional on having consistent connection IDs. If we moved to a scheme that required consistent connection IDs, anyone using that scheme could be required to avoid coalescing different connection IDs. So this seems more of a theoretical concern. And - in anticipation of maybe choosing to make senders use a consistent connection ID - if you might be generating coalesced datagrams with different connection IDs in packets, how hard would it be to ensure that these are consistent? Lars and Ian, I think both of you indicated that you might do this.

Related discussion:
text: carry this connection ID for the duration of the connection or until its peer invalidates the connection ID via a RETIRE_CONNECTION_ID frame (frame-retire-connection-id).  Connection IDs that are issued <del> and not retired are considered active; any active connection ID can be used. </del> <ins> and not retired are considered active; any active connection ID is valid for use at any time, in any packet type.  This includes the connection ID issued by the server via the preferred_address transport parameter. </ins> An endpoint SHOULD ensure that its peer has a sufficient number of available and unused connection IDs.  Endpoints store received