Project: Pipy (Orchestration & Management / Service Proxy)
File: quic-fifm.md

QUIC Frame-in-Flight Management =============================== The QUIC frame-in-flight manager is responsible for tracking frames which were sent which need to be regenerated if the packets they were placed into are designated as lost by the ACK manager. The ACK manager works on the level of packets, whereas the QUIC frame-in-flight manager (FIFM) works on the level of frames. The FIFM comprises three components, collectively known as the FIFM: the Control Frame Queue (CFQ); the Transmitted Packet Information Manager (TXPIM); and the Frame-in-Flight Dispatcher (FIFD). These are introduced in turn below, but first we discuss the various QUIC frame types to establish the need for each component. Analysis of QUIC Frame Retransmission Requirements -- Standard QUIC uses the following frame types: ```plain HANDSHAKE_DONE GCR / REGEN MAX_DATA REGEN DATA_BLOCKED REGEN MAX_STREAMS REGEN STREAMS_BLOCKED REGEN NEWCONNECTIONID GCR RETIRECONNECTIONID GCR PATH_CHALLENGE - PATH_RESPONSE - ACK - (non-ACK-eliciting) CONNECTION_CLOSE special (non-ACK-eliciting) NEW_TOKEN GCR CRYPTO GCR or special RESET_STREAM REGEN STOP_SENDING REGEN MAXSTREAMDATA REGEN STREAMDATABLOCKED REGEN STREAM special PING - PADDING - (non-ACK-eliciting) ``` The different frame types require various different ways of handling retransmission in the event of loss: GCR (Generic Control Frame Retransmission): The raw bytes of the encoded frame can simply be sent again. This retransmission system does not need to understand the specific frame type. A simple queue can be used, with each queue entry being an octet string representing an encoded frame. This queue can also be used for initial transmission of GCR frames, not just retransmissions. REGEN (Regenerate): These frames can be marked for dynamic regeneration when a packet containing them is lost. This has the advantage of using up-to-date data at the time of transmission, so is preferred over `GCR` when possible. Special `STREAM`, `CRYPTO`: `STREAM` frame handling is handled as a special case by the QUIC Send Stream Manager. `CRYPTO` frame retransmission can also be handled using a QUIC Send Stream manager. (`CRYPTO` frames could also be handled via GCR, though suboptimally. We choose to use proper send stream management, just as for application data streams.) Some frame types do not need to be retransmitted even if lost (`PING`, `PADDING`, `PATHCHALLENGE`, `PATHRESPONSE`). Special `CONNECTION_CLOSE`: This frame is a special case and is not retransmitted per se. The following requirements are identified: Need for a generic control queue which can store encoded control frames. This control queue will handle both initial transmission and retransmission of most control frames which do not have special requirements. The ability to determine, when the ACK Manager determines that a packet has been acknowledged, lost or discarded: What stream IDs were sent in a packet, and the logical ranges of application data bytes for each (which may not be one contiguous range). This is needed so that the QUIC Send Stream Manager for a given stream can be informed of lost or acked ranges in the stream. The logical ranges of the CRYPTO stream which were sent in the packet (which may not be one contiguous range), for similar reasons. Which stream IDs had a FIN bit set in the packet. This is needed so that the QUIC Send Stream Manager can be informed for a given stream whether a FIN was lost or

What control frames using the GCR strategy were sent in the packet so that they can be requeued (if lost) or released (if acked or discarded). For each type of frame using the REGEN strategy, a flag as to whether that frame type was contained in the packet (so that the flag can be set again if the packet was lost). The Control Frame Queue (CFQ) -- The CFQ (`QUIC_CFQ`) stores encoded frames which can be blindly retransmitted in the event that they are lost. It facilitates the GCR retransmission strategy. One logical CFQ instance will be needed per PN space per connection. As an optimisation, these three CFQ instances per connection are all modelled by a single `QUIC_CFQ` instance. Each frame in the CFQ is a simple opaque byte buffer, which has the following metadata associated with it: An integral priority value, used to maintain priority ordering. The frame type, which is provided by the caller along with the buffer. This can be determined from the encoded frame buffer, but this saves the CFQ's users from needing to decode it. The CFQ itself does not use this value. A state, which is either `NEW` or `TX`. Frames added to the CFQ have the `NEW` state initially. When the frame is transmitted, it is transitioned to the `TX` state. If the packet it was sent in is subsequently lost, it is transitioned back to the `NEW` state. Frames in the `NEW` state participate in a priority queue (the NEW queue) according to their priority and the CFQ's NEW queue can be iterated in priority order by callers. When a packet containing a CFQ item is acknowledged, the CFQ is informed and the CFQ item is released. A free callback provided when the buffer was added to the CFQ is called, providing an opportunity to free or reuse the buffer. Buffers provided to the CFQ as part of a CFQ item must remain allocated for the duration of their membership of the CFQ. The CFQ maintains memory allocation of CFQ items themselves internally. ```c /* QUIC Control Frame Queue Item ============================= * The CFQ item structure has a public and a private part. This structure documents the public part. */ typedef struct quiccfqitemst QUICCFQ_ITEM; struct quiccfqitem_st { /* These fields are not used by the CFQ, but are a convenience to assist the TXPIM in keeping a list of GCR control frames which were sent in a packet. They may be used for any purpose. */ QUICCFQITEM pkt_prev, pkt_next; / All other fields are private; use ossl_quic_cfq_item_ accessors. */ }; / Returns the frame type of a CFQ item. / uint64t osslquiccfqitemgetframetype(QUICCFQ_ITEM *item); / Returns a pointer to the encoded buffer of a CFQ item. / const unsigned char ossl_quic_cfq_item_get_encoded(QUIC_CFQ_ITEM item); / Returns the length of the encoded buffer in bytes. / sizet osslquiccfqitemgetencodedlen(QUICCFQ_ITEM *item); / Returns the CFQ item state, a QUIC_CFQ_STATE_ value. */ int osslquiccfgitemgetstate(QUICCFQ_ITEM *item); / Returns the PN space for the CFQ item. / int osslquiccfgitemgetpnspace(QUICCFQITEM *item); /* QUIC Control Frame Queue ======================== */ typedef struct quiccfqst QUIC_CFQ; QUICCFQ *osslquiccfqnew(void); void osslquiccfqfree(QUICCFQ *cfq); /* Input Side - */ /* Enqueue a frame to the CFQ. encoded points to the opaque encoded

* free_cb is called by the CFQ when the buffer is no longer needed; freecbarg is an opaque value passed to free_cb. * priority determines the relative ordering of control frames in a packet. Higher numerical values for priority mean that a frame should come earlier in a packet. pn_space is a QUIC_PN_SPACE_ value. * On success, returns a QUICCFQITEM pointer which acts as a handle to the queued frame. On failure, returns NULL. * The frame is initially in the TX state, so there is no need to call osslquiccfqmarktx() immediately after calling this function. * The frame type is duplicated as the frame_type argument here, even though it is also encoded into the buffer. This allows the caller to determine the frame type if desired without having to decode the frame. */ typedef void (cfqfreecb)(unsigned char buf, size_t buf_len, void arg); QUICCFQITEM ossl_quic_cfq_add_frame(QUIC_CFQ cfq, uint32_t priority, uint32t pnspace, uint64t frametype, const unsigned char *encoded, sizet encodedlen, cfqfreecb *free_cb, void *freecbarg); /* Effects an immediate transition of the given CFQ item to the TX state. */ void osslquiccfqmarktx(QUICCFQ *cfq, QUICCFQ_ITEM *item); /* Effects an immediate transition of the given CFQ item to the NEW state, allowing the frame to be retransmitted. If priority is not UINT32_MAX, the priority is changed to the given value. */ void osslquiccfqmarklost(QUICCFQ *cfq, QUICCFQ_ITEM *item, uint32_t priority); /* Releases a CFQ item. The item may be in either state (NEW or TX) prior to the call. The QUICCFQITEM pointer must not be used following this call. */ void osslquiccfqrelease(QUICCFQ cfq, QUIC_CFQ_ITEM item); /* Output Side -- */ /* Gets the highest priority CFQ item in the given PN space awaiting transmission. If there are none, returns NULL. */ QUICCFQITEM ossl_quic_cfq_get_priority_head(QUIC_CFQ cfq, uint32t pnspace); /* Given a CFQ item, gets the next CFQ item awaiting transmission in priority order in the given PN space. In other words, given the return value of osslquiccfqgetpriority_head(), returns the next-lower priority item. Returns NULL if the given item is the last item in priority order. */ QUICCFQITEM ossl_quic_cfq_item_get_priority_next(QUIC_CFQ_ITEM item, uint32t pnspace); ``` The Transmitted Packet Information Manager (TXPIM) -- The Transmitted Packet Information Manager (`QUIC_TXPIM`) is responsible for allocating and keeping bookkeeping structures for packets which have been transmitted, but not yet acknowledged, deemed lost or discarded. It is a self-contained memory pool handing out `QUICTXPIMPKT` structures. Each `QUICTXPIMPKT` is a self-contained data structure intended for consumption by the FIFM. The `QUICTXPIMPKT` structure can be used for: Keeping track of all GCR control frames which were transmitted in each packet, via a linked list of `QUICCFQITEM`s. Keeping track of all REGEN-strategy control frame types, via a flag for each frame type indicating whether the packet contained such a frame. Keeping track of all stream IDs sent in a given packet, and what ranges of the logical stream were sent, and whether a FIN was sent. Keeping track of what logical ranges of the CRYPTO stream were sent. In order to avoid unnecessary allocations, the FIFM also incorporates the ACK Manager's `QUICACKMTX_PKT` structure into its per-packet bookkeeping structure. The intention is for the `QUICTXPIMPKT` to be the principal allocation made per transmitted

The TX packetiser will obtain a `QUICTXPIMPKT` structure from the TXPIM, fill in the structure including the ACK Manager data, and submit it via the FIFD which we introduce below. The TXPIM does do not anything with the `QUICTXPIMPKT` structure itself other than managing its allocation and manipulation. Constructive use of the data kept in the TXPIM is made by the FIFD. ```c /* QUIC Transmitted Packet Information Manager =========================================== */ typedef struct quictxpimst QUIC_TXPIM; typedef struct quictxpimpkt_st { / ACKM-specific data. Caller should fill this. / QUICACKMTXPKT ackmpkt; / Linked list of CFQ items in this packet. / QUICCFQITEM *retx_head; / Reserved for FIFD use. / QUIC_FIFD *fifd; / Regenerate-strategy frames. / unsigned int hadhandshakedone : 1; unsigned int hadmaxdata_frame : 1; unsigned int hadmaxstreamsbidiframe : 1; unsigned int hadmaxstreamsuniframe : 1; unsigned int hadackframe : 1; / Private data follows. / } QUICTXPIMPKT; / Represents a range of bytes in an application or CRYPTO stream. / typedef struct quictxpimchunk_st { / The stream ID, or UINT64_MAX for the CRYPTO stream. / uint64t streamid; /* The inclusive range of bytes in the stream. Exceptionally, if end < start, designates a frame of zero length (used for FIN-only frames). */ uint64_t start, end; /* Whether a FIN was sent for this stream in the packet. Not valid for CRYPTO stream. */ unsigned int has_fin : 1; } QUICTXPIMCHUNK; QUICTXPIM *osslquictxpimnew(void); void osslquictxpimfree(QUICTXPIM *txpim); /* Allocates a new QUICTXPIMPKT structure from the pool. Returns NULL on failure. The returned structure is cleared of all data and is in a fresh initial state. */ QUICTXPIMPKT ossl_quic_txpim_pkt_alloc(QUIC_TXPIM txpim); /* Releases the TXPIM packet, returning it to the pool. */ void osslquictxpimpktrelease(QUICTXPIM *txpim, QUICTXPIM_PKT *fpkt); / Clears the chunk list of the packet, removing all entries. / void osslquictxpimpktclearchunks(QUICTXPIM_PKT *fpkt); / Appends a chunk to the packet. The structure is copied. / int osslquictxpimpktappendchunk(QUICTXPIM_PKT *fpkt, const QUICTXPIMCHUNK *chunk); / Adds a CFQ item to the packet by prepending it to the retx_head list. / void osslquictxpimpktaddcfqitem(QUICTXPIMPKT *fpkt, QUICCFQITEM *item); /* Returns a pointer to an array of stream chunk information structures for the given packet. The caller must call osslquictxpimpktgetnumchunks() to determine the length of this array. * The chunks are sorted by (stream_id, start) in ascending order. */ const QUICTXPIMCHUNK ossl_quic_txpim_pkt_get_chunks(QUIC_TXPIM_PKT fpkt); /* Returns the number of entries in the array returned by osslquictxpimpktget_chunks(). */ sizet osslquictxpimpktgetnumchunks(QUICTXPIM_PKT *fpkt); /* Returns the number of QUICTXPIMPKTs allocated by the given TXPIM that have yet to be returned to the TXPIM. */ sizet osslquictxpimgetinuse(QUIC_TXPIM *txpim); ``` The Frame-in-Flight Dispatcher (FIFD) Finally, the CFQ, TXPIM and some interfaces to the ACKM are tied together via the FIFD (`QUIC_FIFD`). The FIFD is completely stateless and provides reasonable implementations for the on-loss, on-acked and on-discarded callbacks issued by the ACK Manager. The FIFD is used by obtaining a packet structure from the TXPIM, filling it in, and then calling `osslquicfifdpktcommit()`. The FIFD submits the packet to the ACK Manager as a transmitted packet and provides its own callback implementations to the ACK Manager for the packet. Note that the `QUICTXPIMPKT` is returned to the free pool once any of these callbacks occur; once a packet's fate is known (acked, lost or discarded), use is immediately made of the information in the `QUICTXPIMPKT` and the `QUICTXPIMPKT` is immediately released. CFQ items may be freed (on ACK or discard) or transitioned back to the NEW state (on

The FIFD consumes various dependencies so that it can inform the appropriate subsystems in the event of a packet being acked, lost or discarded. In particular: It references a CFQ used to manage CFQ items; It references an ACK manager which it informs of transmitted packets; It references a TXPIM which manages each `QUICTXPIMPKT`; It is provided with a callback to obtain a QUIC Send Stream based on a stream ID. Thus the caller of the FIFD may implement whatever strategy it likes to map stream IDs to QUIC Send Stream instances. It is provided with a callback which is called when it thinks a frame should be regenerated using the REGEN strategy. Some of these are specific to a given stream, in which case a stream ID is specified. All of the state is in the dependencies referenced by the FIFD. The FIFD itself simply glues all of these parts together. ```c typedef struct quicfifdst { / (internals) / } QUIC_FIFD; int osslquicfifdinit(QUICFIFD *fifd, QUIC_CFQ *cfq, QUIC_ACKM *ackm, QUIC_TXPIM *txpim, / stream_id is UINT64_MAX for the crypto stream / OSSLQSS *(*getqssbyid)(uint64t streamid, void *arg), void *getqssbyidarg, / stream_id is UINT64_MAX if not applicable / void (*regenframe)(uint64t frame_type, uint64t streamid, void *arg), void *regenframearg); void osslquicfifdcleanup(QUICFIFD fifd); / (no-op) */ int osslquicfifdpktcommit(QUICFIFD *fifd, QUICTXPIM_PKT *pkt); ``` Typical Intended TX Packetiser Usage TX Packetiser maintains flags for each REGEN-strategy frame type. It sets this flag when the regenerate callback is issued by the FIFD and clears it when transmitting a packet containing such a frame. TX Packetiser obtains a `QUICTXPIMPKT` structure by calling `osslquictxpimpktalloc()`. TX Packetiser fills in the ACKM part of the `QUICTXPIMPKT` (`QUICACKMTX_PKT`), except for the callback fields, which are handled by the FIFD. TX Packetiser queries the ACK Manager to determine if an ACK frame is desired, and if so adds it to the packet. TX Packetiser queries the CFQ to determine what control frames it places in a packet. It does this before adding STREAM or CRYPTO frames (i.e., all CFQ frames are considered of higher priority). For each such frame it places in a packet, it: calls `osslquictxpimpktaddcfqitem()` on the TXPIM to log the CFQ item as having been transmitted in the given packet, so that the CFQ item can be released or requeued depending on the ultimate fate of the packet. For each STREAM or CRYPTO frame included in a packet, the TX Packetiser: informs the QUIC Send Stream instance for that stream that a range of bytes has been transmitted; also informs the QUIC Send Stream instance if FIN was set on a STREAM frame. calls `osslquictxpimpktappend_chunk()` to log a logical range of the given application or crypto stream as having been sent, so that it can be subsequently marked as acknowledged or lost depending on the ultimate fate of the packet. TX Packetiser calls `osslquicfifdpktcommit()`. The FIFD takes care of submitting the packet to the ACK Manager and provides its own callback implementation. It also takes care of informing the CFQ that any CFQ items which were added via `osslquictxpimpktaddcfqitem()` have been transmitted. In the event of packet loss, ACK or discard, the appropriate QUIC Send Stream, CFQ and regenerate callback calls are made. Regardless of the outcome, the TXPIM is released.