Draft: draft-ietf-quic-qlog-main-schema

Commentary: This PR bundles the TypeScript - to - CDDL changes for draft 0.4 for the main schema document ( see other PRs for the QUIC and H3 / QPACK documents ) . List of intended changes : [ x ] Replace TypeScript with CDDL descriptions [ x ] Split up CDDL definitions and JSON examples [ ] Properly explain the $  extension point + include trigger in the example [ ] Properly discuss how event docs should define triggers in the CDDL [ x ] Replace TypeScript explanation in and with CDDL and I - JSON references , basic explanations and considerations . [ x ] Refer to main schema CDDL introduction in H3 and QUIC docs as well [ x ] Explain the need for custom  and hexstring CDDL types [ x ] Add uint32 ,  and uint8 types [ x ] Add $  listings in the QUIC and H3 docs as well ! [ x ] Update QUIC and H3 docs to use uint32 instead of normal uint
Paragraph (version A):
While the qlog schema's are format-agnostic, for readability the qlog
documents will use a JSON-inspired format (RFC8259) for examples and
definitions.
Paragraph (version A):
As qlog can be serialized both textually but also in binary, we
employ a custom datatype definition language, inspired loosely by the
"TypeScript" language [3].
Paragraph (version A):
This document describes how to employ JSON (RFC8259) and JSON Text
Sequences (RFC7464) as textual serializations for qlog in concrete-
formats.  Other documents will describe how to utilize other concrete
serialization options, though tips and requirements for these are
also listed in this document (concrete-formats).
Paragraph (version A):
All timestamps and time-related values (e.g., offsets) in qlog are
logged as doubles in the millisecond resolution.
Paragraph (version A):
Other qlog documents can define their own data types (e.g.,
separately for each Packet type that a protocol supports).
Paragraph (version B):
To define events and data structures, all qlog documents use the
Concise Data Definition Language CDDL.  This document uses the basic
syntax, the specific "text", "uint", "float32", "float64", "bool",
and "any" types, as well as the ".default", ".size", and ".regexp"
control operators, the "~" unwrapping operator, and the "$" extension
point syntax from CDDL.
Paragraph (version B):
"* text => any" : special syntax to indicate 0 or more fields that
have a string key that maps to any value.  Used to indicate a
generic JSON object.
Paragraph (version B):
All timestamps and time-related values (e.g., offsets) in qlog are
logged as "float64" in the millisecond resolution.
Paragraph (version B):
Other qlog documents can define their own CDDL-compatible (struct)
types (e.g., separately for each Packet type that a protocol
supports).
Paragraph (version B):
While the qlog schemas are format-agnostic, and can be serialized in
many ways (e.g., JSON, CBOR, protobuf, ...), this document only
describes how to employ JSON, its subset I-JSON, and its streamable
derivative JSON-Text-Sequences as textual serialization options.  As
such, examples are provided in JSON.  Other documents may describe
how to utilize other concrete serialization options, though tips and
requirements for these are also listed in this document (concrete-
formats).
Paragraph (version A):
there have been several previously broadly deployed qlog versions
based on older drafts of this document (see draft-marx-qlog-main-
schema).  The old values for the "qlog_version" field were "draft-
00", "draft-01" and "draft-02".  When qlog was moved to the QUIC
working group, we decided to switch to a new versioning scheme
which is independent of individual draft document numbers.
However, we do start from 0.3, as conceptually 0.0, 0.1 and 0.2
can map to draft-00, draft-01 and draft-02.
Paragraph (version B):
there have been several previously broadly deployed qlog versions
based on older drafts of this document (see draft-marx-qlog-main-
schema).  The old values for the "qlog_version" field were "draft-
00", "draft-01" and "draft-02".  When qlog was moved to the QUIC
working group, we decided to switch to a new versioning scheme
which is independent of individual draft document numbers.
However, we did start from 0.3, as conceptually 0.0, 0.1 and 0.2
can map to draft-00, draft-01 and draft-02.
Paragraph (version A):
An example of the qlog file's top-level structure is shown in top-
element.
Paragraph (version B):
An example of the qlog file's top-level structure is shown in qlog-
file-def.
Paragraph (version A):
As the summary field is highly deployment-specific, this document
does not specify any default fields or their semantics.  Some
examples of potential entries are shown in summary-example.
Paragraph (version B):
As the summary field is highly deployment-specific, this document
does not specify any default fields or their semantics.  Some
examples of potential entries are shown in summary.
Paragraph (version A):
The "traces" array can thus contain both normal traces (for the
definition of the Trace type, see trace), but also "error" entries.
These indicate that we tried to find/convert a file for inclusion in
the aggregated qlog, but there was an error during the process.
Rather than silently dropping the erroneous file, we can opt to
explicitly include it in the qlog file as an entry in the "traces"
array, as shown in trace-error.
Paragraph (version B):
The "traces" array can thus contain both normal traces (for the
definition of the Trace type, see trace), but also "error" entries.
These indicate that we tried to find/convert a file for inclusion in
the aggregated qlog, but there was an error during the process.
Rather than silently dropping the erroneous file, we can opt to
explicitly include it in the qlog file as an entry in the "traces"
array, as shown in trace-error-def.
Paragraph (version A):
The exact conceptual definition of a Trace can be fluid.  For
example, a trace could contain all events for a single connection,
for a single endpoint, for a single measurement interval, for a
single protocol, etc.  As such, a Trace container contains some
metadata in addition to the logged events, see trace-container.
Paragraph (version B):
The exact conceptual definition of a Trace can be fluid.  For
example, a trace could contain all events for a single connection,
for a single endpoint, for a single measurement interval, for a
single protocol, etc.  As such, a Trace container contains some
metadata in addition to the logged events, see trace-def.
Paragraph (version A):
Two examples from the qvis toolset [4] are shown in qvis-config.
Paragraph (version A):
The vantage_point field describes the vantage point from which the
trace originates, see vantage-point-example.  Each trace can have
only a single vantage_point and thus all events in a trace MUST BE
from the perspective of this vantage_point.  To include events from
multiple vantage_points, implementers can for example include
multiple traces, split by vantage_point, in a single qlog file.
Paragraph (version B):
Two examples from the qvis toolset [3] are shown in qvis-config.
Paragraph (version B):
The vantage_point field describes the vantage point from which the
trace originates, see vantage-point-def.  Each trace can have only a
single vantage_point and thus all events in a trace MUST BE from the
perspective of this vantage_point.  To include events from multiple
vantage_points, implementers can for example include multiple traces,
split by vantage_point, in a single qlog file.
Paragraph (version A):
An example of a qlog event with its component fields is shown in
event-definition.
Paragraph (version B):
An example of a qlog event with its component fields is shown in
event-def.
Paragraph (version B):
The "time" field indicates the timestamp at which the event occured.
Its value is typically the Unix timestamp since the 1970 epoch
(number of milliseconds since midnight UTC, January 1, 1970, ignoring
leap seconds).  However, qlog supports two more succint timestamps
formats to allow reducing file size.  The employed format is
indicated in the "time_format" field, which allows one of three
values: "absolute", "delta" or "relative".
Paragraph (version A):
The first option is good for stateless loggers, the second and third
for stateful loggers.  The third option is generally preferred, since
it produces smaller files while being easier to reason about.  An
example for each option can be seen in time-examples.
Paragraph (version B):
The first option is good for stateless loggers, the second and third
for stateful loggers.  The third option is generally preferred, since
it produces smaller files while being easier to reason about.  An
example for each option can be seen in time-format-ex.
Paragraph (version A):
As such, the default approach in qlog is to concatenate both field
values using the ":" character in the "name" field, as can be seen in
name-example.  As such, qlog category and type names MUST NOT include
this character.
Paragraph (version B):
As such, the default approach in qlog is to concatenate both field
values using the ":" character in the "name" field, as can be seen in
name-ex.  As such, qlog category and type names MUST NOT include this
character.
Paragraph (version A):
One purely illustrative example for a QUIC "packet_sent" event is
shown in data-example.
Paragraph (version B):
This field is defined here as a CDDL extension point (a "socket" or
"plug") named "$ProtocolEventBody".  Other documents MUST properly
extend this extension point when defining new data field content
options to enable automated validation of aggregated qlog schemas.
Paragraph (version B):
The only common field defined for the data field is the "trigger"
field, which is discussed in trigger-field.
Paragraph (version A):
In qlog, the optional "trigger" field contains a string value
describing the reason (if any) for this event instance occuring.
While this "trigger" field could be a property of the qlog Event
itself, it is instead a property of the "data" field instead.  This
choice was made because many event types do not include a trigger
value, and having the field at the Event-level would cause overhead
in some serializations.  Additional information on the trigger can be
added in the form of additional member fields of the "data" field
value, yet this is highly implementation-specific, as are the trigger
field's string values.
Paragraph (version A):
One purely illustrative example of some potential triggers for QUIC's
"packet_dropped" event is shown in trigger-example.
Paragraph (version B):
In qlog, the optional "trigger" field contains a string value
describing the reason (if any) for this event instance occuring, see
data-field.  While this "trigger" field could be a property of the
qlog Event itself, it is instead a property of the "data" field
instead.  This choice was made because many event types do not
include a trigger value, and having the field at the Event-level
would cause overhead in some serializations.  Additional information
on the trigger can be added in the form of additional member fields
of the "data" field value, yet this is highly implementation-
specific, as are the trigger field's string values.
Paragraph (version A):
As such, to provide consistency and ease of tooling in cross-protocol
and cross-context setups, qlog instead defines the common "group_id"
field, which contains a string value.  Implementations are free to
use their preferred string serialization for this field, so long as
it contains a unique value per logical group.  Some examples can be
seen in group-id-example.
Paragraph (version B):
As such, to provide consistency and ease of tooling in cross-protocol
and cross-context setups, qlog instead defines the common "group_id"
field, which contains a string value.  Implementations are free to
use their preferred string serialization for this field, so long as
it contains a unique value per logical group.  Some examples can be
seen in group-id-ex.
Paragraph (version A):
To reduce file size and making logging easier, qlog uses the
"common_fields" list to indicate those fields and their values that
are shared by all events in this component trace.  This prevents
these fields from being logged for each individual event.  An example
of this is shown in common-fields-example.
Paragraph (version B):
To reduce file size and making logging easier, qlog uses the
"common_fields" list to indicate those fields and their values that
are shared by all events in this component trace.  This prevents
these fields from being logged for each individual event.  An example
of this is shown in common-fields-ex.
Paragraph (version A):
The RawInfo:data field can be truncated for privacy or security
purposes (for example excluding payload data).  In this case, the
length properties should still indicate the non-truncated lengths.
Paragraph (version B):
The RawInfo:data field can be truncated for privacy or security
purposes (for example excluding payload data), see truncated-
values.  In this case, the length properties should still indicate
the non-truncated lengths.
Paragraph (version A):
This being said, the authors prefer JSON as the basis for storing
qlog, as it retains full flexibility and maximum interoperability.
Storage overhead can be managed well in practice by employing
compression.  For this reason, this document details both how to
practically transform qlog schema definitions to JSON and to the
streamable JSON Text Sequences.  We discuss concrete options to bring
down JSON size and processing overheads in optimizations.
Paragraph (version B):
This being said, the authors prefer JSON as the basis for storing
qlog, as it retains full flexibility and maximum interoperability.
Storage overhead can be managed well in practice by employing
compression.  For this reason, this document details how to
practically transform qlog schema definitions to JSON, its subset
I-JSON, and its streamable derivative JSON-Text-Sequencess.  We
discuss concrete options to bring down JSON size and processing
overheads in optimizations.
Paragraph (version A):
When using normal JSON serialization, the file extension/suffix
SHOULD be ".qlog" and the Media Type (if any) SHOULD be "application/
qlog+json" RFC6839.
Paragraph (version A):
To facilitate this mapping, the qlog documents employ a format that
is close to pure JSON for its examples and data definitions.  Still,
as JSON is not a typed format, there are some practical peculiarities
to observe.
Paragraph (version A):
While JSON has built-in support for integers up to 64 bits in size,
not all JSON parsers do.  For example, none of the major Web browsers
support full 64-bit integers at this time, as all numerical values
(both floating-point numbers and integers) are internally represented
as floating point IEEE 754 [5] values.  In practice, this limits
their integers to a maximum value of 2^53-1.  Integers larger than
that are either truncated or produce a JSON parsing error.  While
this is expected to improve in the future (as "BigInt" support [6]
has been introduced in most Browsers, though not yet integrated into
JSON parsers), we still need to deal with it here.
Paragraph (version A):
When transforming an int64, uint64 or double from qlog to JSON, the
implementer can thus choose to either log them as JSON numbers
(taking the risk of truncation or un-parseability) or to log them as
strings instead.  Logging as strings should however only be
practically needed if the value is likely to exceed 2^53-1.  In
practice, even though protocols such as QUIC allow 64-bit values for
for example stream identifiers, these high numbers are unlikely to be
reached for the overwhelming majority of cases.  As such, it is
probably a valid trade-off to take the risk and log 64-bit values as
JSON numbers instead of strings.
Paragraph (version A):
Tools processing JSON-based qlog SHOULD however be able to deal with
64-bit fields being serialized as either strings or numbers.
Paragraph (version A):
Unlike most binary formats, JSON does not allow the logging of raw
binary blobs directly.  As such, when serializing a byte or
array<byte>, a scheme needs to be chosen.
Paragraph (version A):
To represent qlog bytes in JSON, they MUST be serialized to their
lowercase hexadecimal equivalents (with 0 prefix for values lower
than 10).  All values are directly appended to each other, without
delimiters.  The full value is not prefixed with 0x (as is sometimes
common).  An example is given in json-bytes-example.
Paragraph (version A):
As such, the resulting string will always have an even amount of
characters and the original byte-size can be retrieved by dividing
the string length by 2.
Paragraph (version A):
In some cases, it can be interesting not to log a full raw blob but
instead a truncated value (for example, only the first 100 bytes of
an HTTP response body to be able to discern which file it actually
contained).  In these cases, the original byte-size length cannot be
obtained from the serialized value directly.  As such, all qlog
schema definitions SHOULD include a separate, length-indicating field
for all fields of type array<byte> they specify.  This allows always
retrieving the original length, but also allows the omission of any
raw value bytes of the field completely (e.g., out of privacy or
security considerations).
Paragraph (version A):
To reduce overhead however and in the case the full raw value is
logged, the extra length-indicating field can be left out.  As such,
tools MUST be able to deal with this situation and derive the length
of the field from the raw value if no separate length-indicating
field is present.  All possible permutations are shown by example in
json-bytes-example-two.
Paragraph (version A):
By definition, JSON strings are serialized surrounded by quotes.
Numbers without.
Paragraph (version A):
Most JSON parsers strictly follow the JSON specification.  This
includes the rule that trailing comma's are not allowed.  As it is
frequently annoying to remove these trailing comma's when logging
events in a streaming fashion, tool implementers SHOULD allow the
last event entry of a qlog trace to be an empty object.  This allows
loggers to simply close the qlog file by appending "{}]}]}" after
their last added event.
Paragraph (version A):
Finally, while not specifically required by the JSON specification,
all qlog field names in a JSON serialization MUST be lowercase.
Paragraph (version B):
When using normal JSON serialization, the file extension/suffix
SHOULD be ".qlog" and the Media Type (if any) SHOULD be "application/
qlog+json" per RFC6839.
Paragraph (version B):
While not specifically required by the JSON specification, all qlog
field names in a JSON serialization MUST be lowercase.
Paragraph (version B):
In order to serialize CDDL-based qlog event and data structure
definitions to JSON, the official CDDL-to-JSON mapping defined in
Appendix E of CDDL SHOULD be employed.
Paragraph (version B):
For some use cases, it should be taken into account that not all
popular JSON parsers support the full JSON format.  Especially for
parsers integrated with the JavaScript programming language (e.g.,
Web browsers, NodeJS), users are recommended to stick to a JSON
subset dubbed I-JSON (or Internet-JSON).
Paragraph (version B):
One of the key limitations of JavaScript and thus I-JSON is that it
cannot represent full 64-bit integers in standard operating mode
(i.e., without using BigInt extensions), instead being limited to the
range of "[-(2**53)+1, (2**53)-1]".  In these circumstances,
Appendix E of CDDL recommends defining new CDDL types for int64 and
uint64 that limit their values to this range.
Paragraph (version B):
While this can be sensible and workable for most use cases, some
protocols targeting qlog serialization (e.g., QUIC, HTTP/3), might
require full uint64 variables in some (rare) circumstances.  In these
situations, it should be allowed to also use the string-based
representation of uint64 values alongside the numerical
representation.  Concretely, the following definition of uint64
should override the original and (web-based) tools should take into
account that a uint64 field can be either a number or string.
Paragraph (version B):
For some use cases (e.g., limiting file size, privacy), it can be
necessary not to log a full raw blob (using the "hexstring" type) but
instead a truncated value (for example, only the first 100 bytes of
an HTTP response body to be able to discern which file it actually
contained).  In these cases, the original byte-size length cannot be
obtained from the serialized value directly.
Paragraph (version B):
As such, all qlog schema definitions SHOULD include a separate,
length-indicating field for all fields of type "hexstring" they
specify, see for example raw-info.  This not only ensures the
original length can always be retrieved, but also allows the omission
of any raw value bytes of the field completely (e.g., out of privacy
or security considerations).
Paragraph (version B):
To reduce overhead however and in the case the full raw value is
logged, the extra length-indicating field can be left out.  As such,
tools MUST be able to deal with this situation and derive the length
of the field from the raw value if no separate length-indicating
field is present.  The main possible permutations are shown by
example in truncated-values-ex.
Paragraph (version A):
When using JSON-SEQ serialization, the file extension/suffix SHOULD
be ".sqlog" (for "streaming" qlog) and the Media Type (if any) SHOULD
be "application/qlog+json-seq" RFC8091.
Paragraph (version B):
When using JSON-SEQ serialization, the file extension/suffix SHOULD
be ".sqlog" (for "streaming" qlog) and the Media Type (if any) SHOULD
be "application/qlog+json-seq" per RFC8091.
Paragraph (version A):
For this to work, some qlog definitions have to be adjusted however.
Mainly, events are no longer part of the "events" array in the Trace
object, but are instead logged separately from the qlog "header", as
indicated by the TraceSeq object in json-seq-example.  Additionally,
qlog's JSON-SEQ mapping does not allow logging multiple individual
traces in a single qlog file.  As such, the QlogFile:traces field is
replaced by the singular QlogFileSeq:trace field.  An example can be
seen in json-seq-example.  Note that the "group_id" field can still
be used on a per-event basis to include events from conceptually
different sources in a single JSON-SEQ qlog file.
Paragraph (version B):
For this to work, some qlog definitions have to be adjusted however.
Mainly, events are no longer part of the "events" array in the Trace
object, but are instead logged separately from the qlog "header", as
indicated by the TraceSeq object in trace-seq-def.  Additionally,
qlog's JSON-SEQ mapping does not allow logging multiple individual
traces in a single qlog file.  As such, the QlogFile:traces field is
replaced by the singular QlogFileSeq:trace field, see qlog-file-seq-
def.  An example can be seen in json-seq-ex.  Note that the
"group_id" field can still be used on a per-event basis to include
events from conceptually different sources in a single JSON-SEQ qlog
file.
Paragraph (version B):
In order to serialize all other CDDL-based qlog event and data
structure definitions to JSON-SEQ, the official CDDL-to-JSON mapping
defined in Appendix E of CDDL SHOULD still be employed.
Paragraph (version A):
Both the JSON and JSON-SEQ formatting options described above are
serviceable in general small to medium scale (debugging) setups.
However, these approaches tend to be relatively verbose, leading to
larger file sizes.  Additionally, generalized JSON(-SEQ)
(de)serialization performance is typically (slightly) lower than that
of more optimized and predictable formats.  Both aspects make these
formats more challenging (though still practical [7]) to use in large
scale setups.
Paragraph (version A):
During the development of qlog, we compared a multitude of
alternative formatting and optimization options.  The results of this
study are summarized on the qlog github repository [8].  The rest of
this section discusses some of these approaches implementations could
choose and the expected gains and tradeoffs inherent therein.  Tools
SHOULD support mainly the compression options listed in compression,
as they provide the largest wins for the least cost overall.
Paragraph (version B):
Both the JSON and JSON-SEQ formatting options described above are
serviceable in general small to medium scale (debugging) setups.
However, these approaches tend to be relatively verbose, leading to
larger file sizes.  Additionally, generalized JSON(-SEQ)
(de)serialization performance is typically (slightly) lower than that
of more optimized and predictable formats.  Both aspects make these
formats more challenging (though still practical [4]) to use in large
scale setups.
Paragraph (version B):
During the development of qlog, we compared a multitude of
alternative formatting and optimization options.  The results of this
study are summarized on the qlog github repository [5].  The rest of
this section discusses some of these approaches implementations could
choose and the expected gains and tradeoffs inherent therein.  Tools
SHOULD support mainly the compression options listed in compression,
as they provide the largest wins for the least cost overall.
Paragraph (version A):
The second option is to replace field values and/or names with
indices into a (dynamic) lookup table.  This is a common compression
technique and can provide significant file size reductions (up to 50%
in our tests, 100MB to 50MB).  However, this approach is even more
difficult to implement efficiently and requires either including the
(dynamic) table in the resulting file (an approach taken by for
example Chromium's NetLog format [9]) or defining a (static) table
up-front and sharing this between implementations.  Implementations
using this approach should not employ a separate file extension (as
it still uses JSON), but rather employ a new value of
"JSON.dictionary" (or "JSON-SEQ.dictionary") for the "qlog_format"
field (see top-level).
Paragraph (version B):
The second option is to replace field values and/or names with
indices into a (dynamic) lookup table.  This is a common compression
technique and can provide significant file size reductions (up to 50%
in our tests, 100MB to 50MB).  However, this approach is even more
difficult to implement efficiently and requires either including the
(dynamic) table in the resulting file (an approach taken by for
example Chromium's NetLog format [6]) or defining a (static) table
up-front and sharing this between implementations.  Implementations
using this approach should not employ a separate file extension (as
it still uses JSON), but rather employ a new value of
"JSON.dictionary" (or "JSON-SEQ.dictionary") for the "qlog_format"
field (see top-level).
Paragraph (version A):
Still, qlog implementers are free to define other qlog formats
depending on their needs and context of use.  These formats should be
described in their own documents, the discussion in this document
mainly acting as inspiration and high-level guidance.  Implementers
are encouraged to add concrete qlog formats and definitions to the
designated public repository [11].
Paragraph (version B):
Still, qlog implementers are free to define other qlog formats
depending on their needs and context of use.  These formats should be
described in their own documents, the discussion in this document
mainly acting as inspiration and high-level guidance.  Implementers
are encouraged to add concrete qlog formats and definitions to the
designated public repository [8].
Paragraph (version A):
As such, when defining a new qlog serialization format or wanting to
utilize qlog-compatible tools with existing codebases lacking qlog
support, it is recommended to define and provide a concrete mapping
from one format to default JSON-serialized qlog.  Several of such
mappings exist.  Firstly, [pcap2qlog]((https://github.com/quiclog/
pcap2qlog) transforms QUIC and HTTP/3 packet capture files to qlog.
Secondly, netlog2qlog [12] converts chromium's internal dictionary-
encoded JSON format to qlog.  Finally, quictrace2qlog [13] converts
the older quictrace format to JSON qlog.  Tools can then easily
integrate with these converters (either by incorporating them
directly or for example using them as a (web-based) API) so users can
provide different file types with ease.  For example, the qvis [14]
toolsuite supports a multitude of formats and qlog serializations.
Paragraph (version B):
As such, when defining a new qlog serialization format or wanting to
utilize qlog-compatible tools with existing codebases lacking qlog
support, it is recommended to define and provide a concrete mapping
from one format to default JSON-serialized qlog.  Several of such
mappings exist.  Firstly, [pcap2qlog]((https://github.com/quiclog/
pcap2qlog) transforms QUIC and HTTP/3 packet capture files to qlog.
Secondly, netlog2qlog [9] converts chromium's internal dictionary-
encoded JSON format to qlog.  Finally, quictrace2qlog [10] converts
the older quictrace format to JSON qlog.  Tools can then easily
integrate with these converters (either by incorporating them
directly or for example using them as a (web-based) API) so users can
provide different file types with ease.  For example, the qvis [11]
toolsuite supports a multitude of formats and qlog serializations.