Draft: draft-ietf-quic-qlog-h3-events

Commentary: and . Partially 
 Looks good to me modulo a few nits and editorials 
 Currently , we have a mechanism for tracking QUIC packet coalescing in a single UDP datagram by means of the field . This was originally as per . Main intended use : when a UDP payload is received containing multiple coalesced QUIC packets , those QUIC packets are given the same " datagram ID " ( which is a receiver - side chosen number that 's unique per received datagram , which is needed because UDP datagrams do n't have a packet number or similar ) . As such , when 2 separate packets are logged in 2 separate events , but they have the same value for the field , it 's clear they were coalesced . Similarly , 2 events with the same indicate sender - side coalescing . As the original issue indicates , this can be deduced from the events as well , but those are often too chatty to leave on . The fields on the events are for completeness / ability to track coalesced packets across the different events . I 'm personally not the biggest fan of this solution ( did you eventually implement this in quic - go , but I do n't immediately see a better approach . CC as he indicated the use of was n't clear to him , which means we either need to revisit or clarify its use . 
 I 'll try to summarise my understanding . The is specified for the following events packetsent packetreceived packetbuffered packetdropped datagramssent ( an array here , but same purpose ) datagramsreceived ( an array here , but same purpose ) The use of the ID is given in and currently says : I do n't mind the motivating use case . I 'm not sure exactly how my implementations could use it for packet sending events because we generate the packet and qlog it before we get into UDP datagram realms . Changes might address that but I 'm not sure it 's worth overhead . Other implementations might find it far more natural to use though . The main part I find confusing is the part about . This is vague and quite specific at the same time . How unique does something need to be ? Can I just use a counter like datagramid = 0 , datagramid=1 for the packets sent on a connection ? Or is the expectation that the identifier is more like a UUID , so that packets are uniquely traceable ? Maybe it depends if we are capturing single connections or traces that span mutliple things , so perhaps a reword to state the requirements and expectations of the datagramid value could help . E.g. " Since QUIC traces may contain many UDP events , unique identifiers are required for each datagram in order to avoid them being confused . Identifiers are not exchanged on the wire , they are only relevant to the logging endpoint . Selecting identifier values is left to implementations that should consider how to generate unique values within the scope of the traces they generate " 
 Currently , most events use , but some still use a separate + setup . out why ( probably just forgot to update all when making the change ) to where appropriate , or add comment why is n't used 
 Currently , we re - use the RawInfo struct to give a simple re - usable way to log raw packet / frame / datagram contents . In the struct , we have two length - fields : the full and , assuming the header length can be calculated from those two . However , we do not have a way to log e.g. , only the payload without the header ( we can only log the full value truncated , but not indicate it starts at the payload ) . This is n't a problem for QUIC / H3 packets / frames , but as pointed out , it is a problem for UDP datagrams . In most stacks , you do n't get the pure UDP packet header and so ca n't log it ( so you 'd always start with the payload ) . This can be solved in several different ways : usage of RawInfo for datagrams explicit that datagram - related events start at the payload a fixed - value / fixed - size dummy UDP datagram ( e.g. , 8 bytes of zeroes ) RawInfo to make it more flexible I 'm currently most in favor of 2 , as I do n't see many people logging full UDP datagrams instead of logging the QUIC packets . 
 Fixed by 
 Currently , we abstract out a lot of the on - the - wire specifics . A good example is in the STREAM frame : there , the length and offset fields are optional , depending on how the frame should be handled . Simply having the same fields optional in qlog does n't convey quite the same semantics : did the implementation simply not log them or were they not present from the start ? More importantly though : if no length is set , the frame extends to the end of the packet , so it does have a length , which you 'd probably want to log ( that 's the way it 's currently designed ) , but so you loose the info that the length field was n't encoded on the wire . This is just one example of similar problems across the board . I rather like the simplicity of the current setup , but we should have ways to signal the explicit wire image as well . cc 
 I can see three options : a more fine - grained , optional field that encodes the actual stream type number instead of the conceptual name we have in frame_type ( like how UnknownFrame works today ) people to log the ( partial ) ( STREAM ) frame header in the raw field to allow for later re - parsing nothing Proposal : use 1 . 
 Can you elaborate on 1 more ? By the time my parser informs me I have a STREAM frame , I might have discarded the on - wire bytes 
 This would be an -extra- option to log besides the in case you really need to know if you got 0x08 , 0x09 , 0x0A ... or 0x0F ( at least for STREAM , for ACK it 'd be 0x02 or 0x03 ) . This could be parsed if you logged the first few bytes in for example , but having a separate field would be more direct . We have precedent for this in e.g. , . If you do n't have that information available at the point you 're logging but still need it well ... you 'd need to log somewhere closer to the wire ? In the ACK case , it is implicit through the logging of ECN info , for stream it is somewhat for the presence of the FIN bool , but for length and offset , it 's more difficult to know sometimes . This is not crucial information , but could be good to know in some highly specific edge case debugging . Got a bug in pcap2qlog about this just the other day , where I forgot to extrapolate the length field from the QUIC packet length if it was n't present in the STREAM frame itself ( ) 
 Some the the question here is what do you mean by wire image . Should something like be raw bytes underlying the varint ( maintaining the information on how the number was encoded on the wire ) , or the parsed varint . Since a single int , it must be the parsed varint , which is slightly easier . But even today I ca n't populate that because my implementation has discarded the data by the time I come to generate the qlog . Changing the code would boil down to me changing the logging pipeline , parsing the same value twice , and/or storing the data twice just for qlog . I 'll note that for H3 this is even harder , because we have to handle non - atomic varint reads as STREAM data comes . 
 So the idea was : Option 1 : parsed varint ( so literally the 0x08 - 0x0F for STREAM ) Option 2 : raw wire image ( so " unparsed " varint ) I do see the confusion ... probably we should use another word than here , since is intended to also be the parsed varint , but the fields are intended to be pure wire bits . With regards to your implementation not having that info available ... I 'm not sure what you want me to say ... these are optional fields that you can output if needed / available , but not something I would expect many implementations to do ( by default ) . I do n't see a way of solving the original issue here ( representing all the wire image nuances in some form ) without having access to those nuances at the logging location :) Could you elaborate on with an example maybe ? Not sure what you mean there ? 
 The main point is that it 's fairly straightforward to log the unprocessed bytes of a QUIC packet or QUIC frame , that 's because an implementation will have received those in whole and know their length . E.g. Read UDP , everything in the datagram . For things on top of QUIC streams it becomes harder because they can span lots of packets . A streaming parser will possibly not have an entire frame available when it want to log an event . If the proposal is to log the parsed varint , it makes things slightly easier 
 how about an optional field , which can be used in any frame type that spans a range of possible values . Then we 'll update UnknownFrame s / rawframetype / frametypevalue . 
 Fixed by
Paragraph (version A):
This document re-uses the definition of the RawInfo data class from
QLOG-MAIN.
Paragraph (version A):
Note: most of this information can also be inferred by looking at a
stream's id, since id's are strictly partitioned at the QUIC level.
Even so, this event has a "Base" importance because it helps a lot in
debugging to have this information clearly spelled out.
Paragraph (version B):
The stream_type_value field is the numerical value without VLIE
encoding.
Paragraph (version A):
HTTP equivalent to the packet_sent event.  This event is emitted when
the HTTP/3 framing actually happens.  Note: this is not necessarily
the same as when the HTTP/3 data is passed on to the QUIC layer.  For
that, see the "data_moved" event in QLOG-QUIC.
Paragraph (version B):
This event is emitted when the HTTP/3 framing actually happens.  This
does not necessarily coincide with HTTP/3 data getting passed to the
QUIC layer.  For that, see the "data_moved" event in QLOG-QUIC.
Paragraph (version A):
HTTP equivalent to the packet_received event.  This event is emitted
when the HTTP/3 frame is parsed.  Note: this is not necessarily the
same as when the HTTP/3 data is actually received on the QUIC layer.
For that, see the "data_moved" event in QLOG-QUIC.
Paragraph (version A):
Note: in HTTP/3, DATA frames can have arbitrarily large lengths to
reduce frame header overhead.  As such, DATA frames can span many
QUIC packets and can be processed in a streaming fashion.  In this
case, the frame_parsed event is emitted once for the frame header,
and further streamed data is indicated using the data_moved event.
Paragraph (version B):
This event is emitted when the HTTP/3 frame is parsed.  Note: this is
not necessarily the same as when the HTTP/3 data is actually received
on the QUIC layer.  For that, see the "data_moved" event in QLOG-
QUIC.
Paragraph (version B):
HTTP/3 DATA frames can have arbitrarily large lengths to reduce frame
header overhead.  As such, DATA frames can span multiple QUIC
packets.  In this case, the frame_parsed event is emitted once for
the frame header, and further streamed data is indicated using the
data_moved event.
Paragraph (version A):
While qlog is a more high-level logging format, it also allows the
inclusion of most raw wire image information, such as byte lengths
and even raw byte values.  This can be useful when for example
investigating or tuning packetization behavior or determining
encoding/framing overheads.  However, these fields are not always
necessary and can take up considerable space if logged for each
packet or frame.  They can also have a considerable privacy and
security impact.  As such, they are grouped in a separate optional
field called "raw" of type RawInfo (where applicable).
Paragraph (version B):
While qlog is a high-level logging format, it also allows the
inclusion of most raw wire image information, such as byte lengths
and byte values.  This is useful when for example investigating or
tuning packetization behavior or determining encoding/framing
overheads.  However, these fields are not always necessary, can take
up considerable space, and can have a considerable privacy and
security impact (see privacy).  Where applicable, these fields are
grouped in a separate, optional, field named "raw" of type RawInfo.
The exact definition of entities, headers, trailers and payloads
depend on the protocol used.
Paragraph (version B):
The RawInfo:data field can be truncated for privacy or security
purposes, see truncated-values.  In this case, the length and
payload_length fields should still indicate the non-truncated lengths
when used for debugging purposes.
Paragraph (version B):
This document does not specify explicit header_length or
trailer_length fields.  In protocols without trailers, header_length
can be calculated by subtracting the payload_length from the length.
In protocols with trailers (e.g., QUIC's AEAD tag), event definition
documents SHOULD define how to support header_length calculation.
Paragraph (version A):
Note: QUIC itself does not have a concept of a "datagram_id".  This
field is a purely qlog-specific construct to allow tracking how
multiple QUIC packets are coalesced inside of a single UDP datagram,
which is an important optimization during the QUIC handshake.  For
this, implementations assign a (per-endpoint) unique ID to each
datagram and keep track of which packets were coalesced into the same
datagram.  As packet coalescing typically only happens during the
handshake (as it requires at least one long header packet), this can
be done without much overhead.
Paragraph (version B):
Since QUIC implementations rarely control UDP logic directly, the raw
data excludes UDP-level headers in all fields.
Paragraph (version B):
The "datagram_id" is a qlog-specific concept to allow tracking of
QUIC packet coalescing inside UDP datagrams.  Implementations can
assign a per-endpoint unique ID to each datagram, and reflect this in
other events to track QUIC packets through processing steps.
Paragraph (version A):
Note: for more details on "datagram_ids", see transport-
datagramssent.
Paragraph (version A):
When a UDP-level datagram is dropped.  This is typically done if it
does not contain a valid QUIC packet (in that case, use
packet_dropped instead).
Paragraph (version B):
For more details on "datagram_ids", see transport-datagramssent.
Paragraph (version B):
When a UDP-level datagram is dropped.  This is typically done if it
does not contain a valid QUIC packet.  If it does, but the QUIC
packet is dropped for other reasons, packet_dropped (transport-
packetdropped) should be used instead.
Paragraph (version A):
Note: we do not for example use a "direction" field (with values "up"
and "down") to specify the data flow.  This is because in some
optimized implementations, data might skip some individual layers.
Additionally, using explicit "from" and "to" fields is more flexible
and allows the definition of other conceptual "layers" (for example
to indicate data from QUIC CRYPTO frames being passed to a TLS
library ("security") or from HTTP/3 to QPACK ("qpack")).
Paragraph (version A):
Note: this event type is part of the "transport" category, but really
spans all the different layers.  This means we have a few leaky
abstractions here (for example, the stream_id or stream offset might
not be available at some logging points, or the raw data might not be
in a byte-array form).  In these situations, implementers can decide
to define new, in-context fields to aid in manual debugging.
Paragraph (version A):
The token carried in an Initial packet can either be a retry token
from a Retry packet, or one originally provided by the server in a
NEW_TOKEN frame used when resuming a connection (e.g., for address
validation purposes).  Retry and resumption tokens typically contain
encoded metadata to check the token's validity when it is used, but
this metadata and its format is implementation specific.  For that,
this field includes a general-purpose "details" field.
Paragraph (version B):
The token carried in an Initial packet can either be a retry token
from a Retry packet, or one originally provided by the server in a
NEW_TOKEN frame used when resuming a connection (e.g., for address
validation purposes).  Retry and resumption tokens typically contain
encoded metadata to check the token's validity when it is used, but
this metadata and its format is implementation specific.  For that,
this event includes a general-purpose "details" field.
Paragraph (version A):
raw_error_code is the actual, numerical code.  This is useful because
some error types are spread out over a range of codes (e.g., QUIC's
crypto_error).
Paragraph (version B):
The error_code_value field is the numerical value without VLIE
encoding.  This is useful because some error types are spread out
over a range of codes (e.g., QUIC's crypto_error).