Draft: draft-ietf-quic-load-balancers

Commentary: This tweak , IMO , substantially improves dynamic allocation . 
 I believe this is good as dynamic SID allocation is going to get , so IMO this is a strict improvement on the current text . However , I 'd like you to review it , as the drawbacks remain significant and if you 're no longer wedded to this approach it might best be removed . It 's true that SID allocation is kind of a pain , but even with dynamic allocation the LB still needs to configure its server pool , so the gains are not all that large . Also adding due to a heinous hack involving zero padding short CIDs , that I ca n't seem to poke a security hole in . 
 thanks , will merge this as a strict improvement ( zero padding is bad but better than " read a bunch of bytes as if was plaintext " ) , then take up the issue of whether to keep dynamic at all . 
 Today , an " lbtimeout " parameter tells LBs how long they need to save an SID allocation after it 's last observed on an incoming packet . Servers may have to retire CIDs when they 're approaching this limit . It 's the only current method of getting rid of these , as we lack any sort of in - band mechanism for the two entities to communicate . Advantages : servers have more SIDs to choose from , which dramatically improves the entropy of PCID ( where , admittedly , the operator has chosen not to care about concealing the mapping ) the server is seldom compelled to retire active CIDs , so the overhead of NEW / RETIRECONNECTIONID frames is low Disadvantages : If the server ID space the number of servers , over time the LB 's table gets very large , to the extent it could challenge the RAM of the device . Careful selection of a server ID length can help , but that 's putting a lot on the operator . The lbtimeout mechanism is annoying because LBs that store a little bit of state have to inspect every packet to make a note of the last time it observed an SID . Servers have to do the same to check when things expire . This seems easy to mess up . Servers also have to decode every incoming Initial CID to extract the SID . An alternative would configure servers to keep a small number of SIDs . This could be as low as 1 but might be 8 or 16 . If an allocatable SID arrives at the server , if it does not instantly use that SID in the CID it sends with the server hello , it forfeits the allocation . Example of acceptance : SID 0x14 arrives in a Client Initial . LB hashes it to a server and adds it to a hash table of provisional allocations Server generates CID using 0x14 and sends it the Server Initial , adding it to its list of server IDs LB notes that the first inbound short header encoded 0x14 and makes it a permanent allocation Example of rejection SID 0x15 arrives in a Client Initial . LB hashes it to a server and adds it to a hash table of provisional allocations Other long headers with 0x15 , with any 4 - tuple , also go to the server thanks to the provisional list ( you ca n't be provisional to two servers ! ) Server generates a CID that encodes 0x14 . LB notes that the first inbound short header did not encode 0x15 and decrements the ref count for the provisional allocation , deleting it if the ref count is zero . We could even make it so the server MUST accept the first n that it can ; then , the LB need not even do a provisional allocation after that peer has reached the threshold . Advantages : much fewer resources for SID tables no timers , less CID parsing & inspection the architecture does n't mandate more CID retiring due to changing SIDs Disadvantages : ? 
 Ca n't make it a MUST for the server to take the first N SIDs ; if it has its own load balancer ( eg routing to multiple cores ) it will have to wait until each of them has gotten an SID , which might take a while . 
 This has a trivial attack : If I send a storm of initial packets with random CIDs , followed by 1RTT packets with the same CID , I force the LB to add it to its table , rapidly filling it up . We could fix this by forcing the LB to sniff SCIDs from the server instead of 1 - RTT packets , which seems a little gross . For not - PCID , it would be sufficient to simply check that there 's a new CID , even if it encodes the same SID . 
 I am kind of convinced that " padding with zeroes " will not end well . I am also a tiny bit worried that relying on the values provided in initial packets for dynamic allocations will backfire when clients become creative . The spec says " When an Initial packet is sent by a client that has not previously received an Initial or Retry packet from the server , the client populates the Destination Connection ID field with an unpredictable value . This Destination Connection ID MUST be at least 8 bytes in length . " But " unpredictable value " is in the eyes of the beholder , and " " is just as unpredictable as " " . I would feel better if the dynamic method was only used in conjunction with stream encryption and length 8 CID , because clients will have a hard time predicting what an arbitrary CID decrypts to : flip one bit in the input , and every output bit has a 50 % chance of splitting . The block cipher might also work , but will definitely require padding . LG , though the more I think about this , the more I want to just move away from this approach . In retrospect , I think it 's just a bit too clever for our own good .
QUIC packets QUIC-TRANSPORT usually contain a connection ID to allow
endpoints to associate packets with different address/ port 4-tuples
to the same connection context.  This feature makes connections
robust in the event of NAT rebinding.  QUIC endpoints usually
designate the connection ID which peers use to address packets.
Server-generated connection IDs create a potential need for out-of-
band communication to support QUIC.
QUIC packets RFC9000 usually contain a connection ID to allow
endpoints to associate packets with different address/port 4-tuples
to the same connection context.  This feature makes connections
robust in the event of NAT rebinding.  QUIC endpoints usually
designate the connection ID which peers use to address packets.
Server-generated connection IDs create a potential need for out-of-
band communication to support QUIC.
Load balancers MUST forward packets with compliant DCIDs to a server
in accordance with the chosen routing algorithm.
All other DCIDs are routable.
Load balancers SHOULD drop short header packets with unroutable
DCIDs.
Load balancers MAY drop packets with long headers and unroutable
DCIDs if and only if it knows that the encoded QUIC version does not
allow an unroutable DCID in a packet with that signature.  For
example, a load balancer can safely drop a QUIC version 1 Handshake
packet with an unroutable DCID, as a version 1 Handshake packet sent
to a QUIC-LB routable server will always have a server-generated
routable CID.  The prohibition against dropping packets with long
headers remains for unknown QUIC versions.
Servers that receive packets with unroutable CIDs MUST use the
available mechanisms to induce the client to use a routable CID in
future packets.  In QUIC version 1, this requires using a routable
CID in the Source CID field of server-generated long headers.
There are conditions described below where a load balancer routes a
packet using a "fallback algorithm."  It can choose any algorithm,
without coordination with the servers, but the algorithm SHOULD be
deterministic over short time scales so that related packets go to
the same server.  The design of this algorithm SHOULD consider the
version-invariant properties of QUIC described in RFC8999 to maximize
its robustness to future versions of QUIC.
A fallback algorithm MUST NOT make the routing behavior dependent on
any bits in the first octet of the QUIC packet header, except the
first bit, which indicates a long header.  All other bits are QUIC
version-dependent and intermediaries SHOULD NOT base their design on
version-specific templates.
For example, one fallback algorithm might convert a unroutable DCID
to an integer and divided by the number of servers, with the modulus
used to forward the packet.  The number of servers is usually
consistent on the time scale of a QUIC connection handshake.  Another
might simply hash the address/port 4-tuple.  See also version-
invariance.
A QUIC-LB configuration MAY significantly over-provision the server
ID space (i.e., provide far more codepoints than there are servers)
to increase the probability that a randomly generated Destination
Connection ID is unroutable.
In the dynamic allocation method, the load balancer assigns server
IDs dynamically so that configuration does not require fixed server
ID assignment.  This reduces linkability.  However, it requires state
at the load balancer that roughly scales with the number of
connections, until the server ID codespace is exhausted.
The configuration agent does not assign server IDs, but does
configure a server ID length and an "LB timeout".  The server ID MUST
be at least one and no more than seven octets.
The load balancer maintains a table of all assigned server IDs and
corresponding routing information, which is initialized empty.  These
tables are independent for each operating configuration.
The load balancer MUST keep track of the most recent observation of
each server ID, in any sort of packet it forwards, in the table and
delete the entries when the time since that observation exceeds the
LB Timeout.
Each server maintains a list of server IDs assigned to it,
initialized empty.  For each SID, it records the last time it
received any packet with an CID that encoded that SID.
In the dynamic allocation method, the load balancer assigns server
IDs dynamically so that configuration does not require fixed server
ID assignment.  This reduces linkability and simplifies
configuration.  However, it also limits the length of the server ID
and requires the load balancer to lie on the path of outbound
packets.  As the server mapping is no longer part of the
configuration, standby load balancers need an out-of-band mechanism
to synchronize server ID allocations in the event of failures of the
primary device.
To summarize, the load balancer forwards incoming Initial packets
arbitrarily and both load balancer and server are sometimes able to
infer a potential server ID allocation from the CID in the packet.
The server can signal acceptance of that allocation by using it
immediately, in which case both entities add it to their permanent
table.  Usually, however, the server will reject the allocation by
not using it, in which case it is not added to the permanent
assignment list.
The configuration agent does not assign server IDs, but does
configure a server ID length.  The server ID MUST be at least one and
no more than seven octets.  See sid-limits for other considerations
if also using the Plaintext CID algorithm.
The load balancer maintains a mapping of assigned server IDs to
routing information for servers, initialized as empty.  This mapping
is independent for each operating configuration.
Note that when the load balancer's tables for a configuration are
empty, all incoming DCIDs corresponding to that configuration are
unroutable by definition.
If the config rotation bits do not match a known configuration,
the load balancer routes the packet using a fallback algorithm
(see fallback-algorithm).  It does not extract a server ID.
If there is a matching configuration, but the CID is not long
enough to apply the algorithm, the load balancer pads the
connection ID with zeros to the required length.
Otherwise, the load balancer extracts the server ID in accordance
with the configured algorithm and parameters.
If the load balancer extracted a server ID already in its mapping, it
routes the packet accordingly.  If the server ID is not in the
mapping, it routes the packet according to a fallback algorithm and
awaits the first long header the server sends in response.
If the load balancer extracted an unassigned server ID and observes
that the first long header packet the server sends has a Source
Connection ID that encodes the same server ID, it adds that server ID
to the mapping.  Otherwise, it takes no action.
Each server maintains a list of server IDs assigned to it,
initialized empty.
If the DCID is not long enough to decode using the configured
algorithm, extract a number of octets equal to the server ID
length, beginning with the second octet.  If the extracted value
does not match a server ID in the server's list, add it to the
list.
If the DCID is long enough to decode but the server ID is not in
the server's list, add it to the list.
After any possible SID is extracted, the server processes the packet
normally.
When a server needs a new connection ID, it uses one of the server
IDs in its list to populate the server ID field of that CID.  It
SHOULD vary this selection to reduce linkability within a connection.
After loading a new configuration or long periods of idleness, a
server may not have any available SIDs.  This is because an incoming
packet may not the config rotation bits necessary to extract a server
ID in accordance with the algorithm above.  When required to generate
a CID under these conditions, the server MUST generate CIDs using the
5-tuple routing codepoint (see config-failover.  Note that these
connections will not be robust to client address changes while they
use this connection ID.  For this reason, a server SHOULD retire
these connection IDs and replace them with routable ones once it
receives a client-generated CID that allows it to acquire a server
ID.  As, statistically, one in every four such CIDs can provide a
server ID, this is typically a short interval.
If the DCID is not long enough to decode using the configured
algorithm, pad it with zeros to the required length and extract a
server ID.
If the DCID is long enough to decode, extract the server ID.
If the server ID is not already in its list, the server MUST decide
whether or not to immediately use it to encode a CID on the new
connection.  If it chooses to use it, it adds the server ID to its
list.  If it does not, it MUST NOT use the server ID in future CIDs.
The server SHOULD NOT use more than one CID, unless it is close to
exhausting the nonces for an existing assignment.  Note also that the
load balancer may observe a single entity claiming multiple server
IDs because that entity actually represents multiple servers devices
or processors.
The server MUST generate a new connection ID if the client-generated
CID is of insufficient length for the configuration.
The server then processes the packet normally.
When a server needs a new connection ID, it uses one of the server
IDs in its list to populate the server ID field of that CID.  It MAY
vary this selection to reduce linkability within a connection.
After loading a new configuration, a server may not have any
available SIDs.  This is because an incoming packet may not contain
the config rotation bits necessary to extract a server ID in
accordance with the algorithm above.  When required to generate a CID
under these conditions, the server MUST generate CIDs using the
5-tuple routing codepoint (see config-failover.  Note that these
connections will not be robust to client address changes while they
use this connection ID.  For this reason, a server SHOULD retire
these connection IDs and replace them with routable ones once it
receives a client-generated CID that allows it to acquire a server
ID.  As, statistically, one in every four such CIDs can provide a
server ID, this is typically a short interval.
If a server has not received a connection ID encoding a particular
server ID within the LB timeout, it MUST retire any outstanding CIDs
that use that server ID and cease generating any new ones.
A server SHOULD have a mechanism to stop using some server IDs if the
list gets large relative to its share of the codepoint space, so that
these allocations time out and are freed for reuse by servers that
have recently joined the pool.
The Plaintext CID Algorithm makes no attempt to obscure the mapping
of connections to servers, significantly increasing linkability.  The
format is depicted in the figure below.
The Block Cipher CID Algorithm, by using a full 16 octets of
plaintext and a 128-bit cipher, provides higher cryptographic
protection and detection of non-compliant connection IDs.  However,
it also requires connection IDs of at least 17 octets, increasing
overhead of client-to-server packets.
The server then encrypts the second through seventeenth octets using
the 128-bit AES-ECB cipher.
Alternatively, servers MAY implement the technique described in
Section 14.4.1 of QUIC-TRANSPORT to increase the likelihood a Source
Connection ID is included in ICMP responses to Path Maximum
Transmission Unit (PMTU) probes.  Load balancers MAY parse the echoed
packet to extract the Source Connection ID, if it contains a QUIC
long header, and extract the Server ID as if it were in a Destination
CID.
Alternatively, servers MAY implement the technique described in
Section 14.4.1 of RFC9000 to increase the likelihood a Source
Connection ID is included in ICMP responses to Path Maximum
Transmission Unit (PMTU) probes.  Load balancers MAY parse the echoed
packet to extract the Source Connection ID, if it contains a QUIC
long header, and extract the Server ID as if it were in a Destination
CID.
If a Retry Service maintains no per-flow state whatsoever, it
cannot distinguish between valid and invalid non_Initial packets
and MUST forward all non-Initial Packets to the server.
If a Retry Service maintains no per-flow state whatsoever, it
cannot distinguish between valid and invalid non-Initial packets
and MUST forward all non-Initial Packets to the server.
The 96 bit unique token number is set to a random value using a
cryptography- grade random number generator.
The associated data is a formatted as a pseudo header by combining
the cleartext part of the token with the IP address of the client.
When in active mode, the service MUST generate Retry tokens with the
format described above when it receives a client Initial packet with
no token.
The server MUST validate all tokens that arrive in Initial packets,
as they may have bypassed the Retry service.
As discussed in QUIC-TRANSPORT, a server MUST NOT send a Retry packet
in response to an Initial packet that contains a retry token.
For Stream Cipher CID Routing, the servers and load balancer also
MUST have a common understanding of the key and nonce length.
For Block Cipher CID Routing, the servers and load balancer also MUST
have a common understanding of the key.
The server ID encodings, and requirements for their handling, are
designed to be QUIC version independent (see QUIC-INVARIANTS).  A
QUIC-LB load balancer will generally not require changes as servers
deploy new versions of QUIC.  However, there are several unlikely
future design decisions that could impact the operation of QUIC-LB.
The server ID encodings, and requirements for their handling, are
designed to be QUIC version independent (see RFC8999).  A QUIC-LB
load balancer will generally not require changes as servers deploy
new versions of QUIC.  However, there are several unlikely future
design decisions that could impact the operation of QUIC-LB.
While this document does not update the commitments in QUIC-
INVARIANTS, the additional assumptions are minimal and narrowly
scoped, and provide a likely set of constants that load balancers can
use with minimal risk of version- dependence.
While this document does not update the commitments in RFC8999, the
additional assumptions are minimal and narrowly scoped, and provide a
likely set of constants that load balancers can use with minimal risk
of version- dependence.
Section 21.9 of QUIC-TRANSPORT discusses the Stateless Reset Oracle
attack.  For a server deployment to be vulnerable, an attacking
client must be able to cause two packets with the same Destination
CID to arrive at two different servers that share the same
cryptographic context for Stateless Reset tokens.  As QUIC-LB
requires deterministic routing of DCIDs over the life of a
connection, it is a sufficient means of avoiding an Oracle without
additional measures.
To avoid this, the configuration agent SHOULD issue QUIC-LB
configurations to mutually distrustful servers that have different
keys for encryption algorithms.  In many cases, the load balancers
can distinguish these configurations by external IP address.
When configuring QUIC-LB, administrators must evaluate the privacy
tradeoff considering the relative value of each of these properties,
given the trust model between tenants, the presence of methods to
obscure the domain name, and value of address migration in the tenant
use cases.
As the plaintext algorithm makes no attempt to conceal the server
mapping, these deployments SHOULD simply use a common configuration.
Section 21.9 of RFC9000 discusses the Stateless Reset Oracle attack.
For a server deployment to be vulnerable, an attacking client must be
able to cause two packets with the same Destination CID to arrive at
two different servers that share the same cryptographic context for
Stateless Reset tokens.  As QUIC-LB requires deterministic routing of
DCIDs over the life of a connection, it is a sufficient means of
avoiding an Oracle without additional measures.
Note also that when a server starts using a new QUIC-LB config
rotation codepoint, new CIDs might not be unique with respect to
previous configurations that occupied that codepoint, and therefore
different clients may have observed the same CID and stateless reset
token.  A straightforward method of managing stateless reset keys is
to maintain a separate key for each config rotation codepoint, and
replace each key when the configuration for that codepoint changes.
Thus, a server transitions from one config to another, it will be
able to generate correct tokens for connections using either type of
CID.
Section 6.6 of QUIC-TLS states that "Endpoints MUST count the number
of encrypted packets for each set of keys.  If the total number of
encrypted packets with the same key exceeds the confidentiality limit
for the selected AEAD, the endpoint MUST stop using those keys."  It
goes on with the specific limit: "For AEAD_AES_128_GCM and
AEAD_AES_256_GCM, the confidentiality limit is 2^23 encrypted
packets; see Appendix B.1."  It is prudent to adopt the same limit
here, and configure the service in such a way that no more than 2^23
tokens are generated with the same key.
Section 6.6 of RFC9001 states that "Endpoints MUST count the number
of encrypted packets for each set of keys.  If the total number of
encrypted packets with the same key exceeds the confidentiality limit
for the selected AEAD, the endpoint MUST stop using those keys."  It
goes on with the specific limit: "For AEAD_AES_128_GCM and
AEAD_AES_256_GCM, the confidentiality limit is 2^23 encrypted
packets; see Appendix B.1."  It is prudent to adopt the same limit
here, and configure the service in such a way that no more than 2^23
tokens are generated with the same key.
When using Dynamic SID allocation, the load balancer's SID table can
be as large as 2^56 entries, which is prohibitively large.  To
constrain the size of this table, servers are encouraged to accept as
few SIDs as possible, so that the remainder do not enter the load
balancer's table.