Code change:
5. <del> The "zstd-d" content encoding identifies a resource that is a </del> <ins> The "dcz" content encoding identifies a resource that is a </ins> "Dictionary-Compressed Zstandard" stream. <del> A "Dictionary-Compressed Zstandard" stream is a Zstandard RFC8878 stream for a response that has been compressed with an external dictionary. </del> <ins> A "Dictionary-Compressed Zstandard" stream is a binary stream that starts with a 40-byte fixed header and is followed by a Zstandard RFC8878 stream of the response that has been compressed with an external dictionary. The 40-byte header consists of a fixed 8-byte sequence followed by the 32-byte SHA-256 hash of the external dictionary that was used to compress the resource: The 40-byte header is a Zstandard skippable frame (little-endian 0x184D2A5E) with a 32-byte length (little-endian 0x00000020) that is compatible with existing Zstandard decoders. </ins> <del> Clients that announce support for zstd-d content encoding MUST be able to decompress resources that were compressed with a window size of at least 8 MB or 1.25 times the size of the dictionary, which ever is greater, up to a maximum of 128 MB. </del> <ins> Clients that announce support for dcz content encoding MUST be able to decompress resources that were compressed with a window size of at least 8 MB or 1.25 times the size of the dictionary, which ever is greater, up to a maximum of 128 MB. </ins> The window size used will be encoded in the content (currently, this can be expressed in powers of two only) and it MUST be lower than

Related discussion:
text: This eliminates the response header and moves the hash of the dictionary into the payload of the response. Specifically, it adds 36-byte header in front of the compressed stream which is a 4 or 8-byte signature followed by the 32-byte SHA-256 hash digest. The signature is different for the two different compressions: Dictionary-Compressed Brotli: (0xFF followed by ) Dictionary-Compressed Zstandard: (32-byte skippable Zstandard frame) This also changes the content-encoding names for the different schemes to make it clear that they are a new format and not a raw brotli or Zstandard stream ( and ). From the cli, the stream is equivalent to: or
FWIW, it doesn't look like it would be particularly difficult to use a skippable frame instead of a custom header in the Zstandard case if that's the way we want to go. It's 4-bytes bigger (for the frame size) so it will essentially have an 8-byte "magic number" (since the size will always be the same). It won't help with the creation of the files but if there's benefit to have the existing tooling be able to decode them (without verifying the hash because it will just skip it) we can go that route. On the client side, it feels like it will be cleaner on the implementation to use the same flow for verifying the dictionary hash and magic number for both encodings without having to deal with different sized magic numbers (and always buffer and remove the first 36 bytes) but it's not a significant difference.
Still LGTM
NAME could you please take a look and make sure I got the byte sequence for the skippable frame correct? I tested with the zstd cli and it was able to decode the file with the header without issue when created like this:
Thanks. Updated and added an example with the zstd cli.
This is a discussion we've had several times when defining content codings, but it seems like there is never really a single answer. Should the content coding be self-describing, or can it rely on metadata in fields? The compression dictionary work uses header fields to identify which compression dictionary is in use. Originally, the client would indicate a dictionary and the server would indicate use of that dictionary by choosing the content coding.