jiebi/CodeConvo huggingface dataset:
Code change:
operators might prefer only integrity protected content be made transparent. <del> 5.1. </del> <ins> 6.1. </ins> In cases where a single merkle root and multiple inclusion paths are used to prove inclusion for multiple payloads.  There is a risk that

Related discussion:
text: There were basically 2 paths discussed: (implemented in this PR) treealg (TBD1): RFC9162 (1) inclusion proof (TBD2): bytes based on treealg consistency proof (TBD3): bytes based on treealg treealg (TBD1): BitcoinMerkleTree (2) inclusion proof (TBD2): bytes based on treealg consistency proof (TBD3): bytes based on treealg treealg (TBD1): Merkle^2 Tree (3) inclusion proof (TBD2): bytes based on treealg consistency proof (TBD3): bytes based on treealg search proof (TBD4): bytes based on treealg non inclusion proof (TBD5): bytes based on treealg (previously discussed at IETF 115) treealg (TBD1) : "RFC9162SHA256INCLUSIONPROOF" (TBDA) proof (TBDB): bytes based on alg treealg (TBD1) : "RFC9162SHA256CONSISTENCYPROOF" (TBDC) proof (TBDB): bytes based on alg Other things to consider... Can an unprotected header contain multiple proofs for a single type? Can an unprotected header contain multiple proofs of multiple types?
NAME NAME thank you for the reviews. All the CDDL examples will need some help, and certainly there is a lot more editorial work that needs to be done to fully align to this approach. I tried to do the minimum in this PR to host the discussion, but if we merge this, there will need to be further cleanup.
I should point out that the deletion of the consistency proof CDDL, means that it shows up before you define it below. If you think that is ok, I am fine with it.Approved, acknowledging the need for further editorial work & CDDL examplesI agree with Orie's choices; "verifiable data structure" and "proof type" are good generalizations, and I prefer the 2-layer algorithm as e.g. some verifiers may care about a commitment to a VDS even if they don't use a particular proof type.
Code change:
3. <del> In the following section, the concepts of SD-JWTs and releases are described at a conceptual level. </del> <ins> In the following section, the concepts of SD-JWTs and SD-JWT Releases are described at a conceptual level. </ins> 3.1.

Related discussion:
text: Fixing the "release" in the sense of a defined term as "SD-JWT Release"
We have fixed the term and definitions but a few still appear in the main text. We need to fix them as well, hopefully before the first individual draft is uploaded. So, s/signatures called releases/signatures called SD-JWT Releases/ s/SD-JWTs and releases/SD-JWTs and SD-JWT Releases/ s/Verifying a Release/Verifying a SD-JWT Release/
Fixed with PR
Code change:
In addition to these mechanisms, the PING frame provides a way for a client to easily test a connection.  Connections that remain idle can <del> become broken as some middleboxes (for instance, network address translators or load balancers) silently discard connection bindings. The PING frame allows a client to safely test whether a connection is still active without sending a request. </del> <ins> become broken, because some middleboxes (for instance, network address translators or load balancers) silently discard connection bindings.  The PING frame allows a client to safely test whether a connection is still active without sending a request. </ins> 8.8.

Related discussion:
text: : Does "as" mean "while" or "because" in this sentence? Original: Connections that remain idle can become broken as some middleboxes (for instance, network address translators or load balancers) silently discard connection bindings.
"As" means "because" in this sentence.
Code change:
This specification updates the HTTP related aspects of the existing registration procedures for message header fields defined in RFC3864. <del> It defines both a new registration procedure and moves HTTP field definitions into a separate registry. </del> <ins> It replaces the old procedures as they relate to HTTP, by defining a new registration procedure and moving HTTP field definitions into a separate registry. </ins> Please create a new registry as outlined in fields.registry.

Related discussion:
text: The follow-up discussion on indicated that the nature of the updates to RFC 3864 is essentially a "partial obsoletes", which means that we do not want to be part of BCP 90 even if we could -- we're carving off a piece of it and branching out on our own. This PR attempts to reword the description slightly to indicate that we're not just tweaking part of RFC 3864, but rather replacing part of it (the parts that relate to HTTP), hopefully without bloating the text too much.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: This eliminates the response header and moves the hash of the dictionary into the payload of the response. Specifically, it adds 36-byte header in front of the compressed stream which is a 4 or 8-byte signature followed by the 32-byte SHA-256 hash digest. The signature is different for the two different compressions: Dictionary-Compressed Brotli: (0xFF followed by ) Dictionary-Compressed Zstandard: (32-byte skippable Zstandard frame) This also changes the content-encoding names for the different schemes to make it clear that they are a new format and not a raw brotli or Zstandard stream ( and ). From the cli, the stream is equivalent to: or
FWIW, it doesn't look like it would be particularly difficult to use a skippable frame instead of a custom header in the Zstandard case if that's the way we want to go. It's 4-bytes bigger (for the frame size) so it will essentially have an 8-byte "magic number" (since the size will always be the same). It won't help with the creation of the files but if there's benefit to have the existing tooling be able to decode them (without verifying the hash because it will just skip it) we can go that route. On the client side, it feels like it will be cleaner on the implementation to use the same flow for verifying the dictionary hash and magic number for both encodings without having to deal with different sized magic numbers (and always buffer and remove the first 36 bytes) but it's not a significant difference.
Still LGTM
NAME could you please take a look and make sure I got the byte sequence for the skippable frame correct? I tested with the zstd cli and it was able to decode the file with the header without issue when created like this:
Thanks. Updated and added an example with the zstd cli.
This is a discussion we've had several times when defining content codings, but it seems like there is never really a single answer. Should the content coding be self-describing, or can it rely on metadata in fields? The compression dictionary work uses header fields to identify which compression dictionary is in use. Originally, the client would indicate a dictionary and the server would indicate use of that dictionary by choosing the content coding.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: This made interpreting the body of the response quite challenging in that you needed to have a request in order to make sense of it. More recently, the specification has changed to having the client list available dictionaries, with the server echoing the one it chooses. Both use header fields. There is a third option, which is to embed the dictionary identification (which is a hash of the dictionary) ahead of the compressed content. This has some real advantages: Requests could use delta encoding. There is - of course - a real question about availability of the dictionary, but we have that problem with encrypted content codings and keys. That is a problem we solve for responses with a field. It might be solved using the same technique, combined with the client-initiated content coding advice. Or applications can use their own systems. A field costs a lot more bits. This would not benefit from header compression, so it might be a net loss in some cases, but in many cases it would be a distinct win because 32 bytes of hash in the body of a request is far less than a field name and field value containing a base64 encoding of those same 32 bytes. It also comes with disadvantages: Header compression could remove a lot of the cost in bytes. It might be ever-so-slightly more complex for encoding and decoding to have to split the first 32 bytes off the body of a message.
On the encoding side, the main downside is that the cli tooling for brotli and Zstandard don't currently do the embedding so tooling would have to be added to prepend the hash to the files in a way that isn't standard for either (yet anyway) and for manually decompressing the files. Zstandard has identifiers for the dictionaries when using non-raw dictionaries but both assume that raw dictionaries will be negotiated out of band. Technically it would be a pretty trivial modification for clients and servers that are doing the work, I'm just a bit concerned about the developer experience changes (and whatever needs to be done to get both brotli and Zstandard to understand what amounts to new file formats).
I opened issues with brotli and ZStandard to see if they would consider adding it to the format of their respective file formats.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: If it's an optional metadata tag that is backwards compatible with existing encoders and decoders I could see it providing quite a bit of value, even in the non-HTTP case of dictionary compression.
There was some discussion in the ZStandard repo about possibly reserving one of the skippable frame magic numbers for embedding a dictionary ID but there's some level of risk for collision with people who may be using those frames for watermarking or other application-specific use cases. As best as I can tell, the brotli stream format doesn't have a similar frame capability for metadata or application-specific data. We could create a container format that held the dictionary ID and stream (header basically, not unlike zip vs deflate) but that feels like a fairly large effort and the tooling would have to catch up to make it easy for developers to work with. At this point I'm hesitant to recommend adding anything to the payload itself that the existing brotli and zstd tooling can't process. Being able to create, fetch and test the raw files is quite useful for the developer workflow and for debugging deployments. Would it make sense to allow for future encoding formats to include a dictionary ID in the file itself and make the header optional in those cases (and make the embedded ID authoritative)? I'm not sure if that would make sense in this draft since this one is limited to the 2 existing encodings and it would be addressed in a new draft when the new encodings are created or if it makes sense to allow for it here without requiring that and explicitly require embedded ID's.
I don't think that you want to change the zstd or brotli format, only the content coding. That is, something like this: This does partly work against the idea that you might have a bunch of files that contain compressed versions of content. You can't just say . And you can't just decompress without some processing. What you get in exchange is a message payload with the advantages I described.
I do think we need to define a file format for it if we go down this path to ease adoption and it should probably have a magic signature at the beginning. Maybe something like gzip is to deflate but with a simple 3-byte signature followed by the hash followed by the stream data.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: Assuming we create a cli tool to do all of the work for compressing/decompressing them, I'll ping several of the current origin trial participants to see how it will fit into their workflow. Something like: Dictionary-compressed file/stream format with a 3-byte signature indicating hash and compression type, followed by dictionary hash, followed by raw compressed stream. Something like for Dictionary-Compressed Brotli with Sha-256 hash and for Dictionary-Compressed Zstandard with Sha-256 hash. Create a cli script (cross-platform, available in npm, bun, apt, etc) that creates the files and (optionally) calls out to brotli and zstd to do the compression. We could include any special ZStandard compression window handling in the format spec as well as the cli tool. I'm assuming something like this would be better off as it's own draft that this references or do you think it makes sense to define it here? I agree there are significant benefits to having the hash paired directly with the resource. I just want to be careful to make sure whatever we do fits in well with the developer workflow as well.
Adding the header would add some work to typical CI workflow. At least in my case, the diff file stream is created using the cli using the options. Adding the header would require creating a new file with the header and add the generated stream. If this can be done in a published script it would simplify the logic (FYI - URL currently does yet not support brotli bindings, see URL) but not a requirement IMO. Do I understand correctly that with this idea implemented the header would not be necessary anymore? That would simplify some CDN configurations which currently needs to add it. Related to this - what in this case would be header value? current spec is it should be which I understand as brotli dictionary stream. If we add the header, how would a client be able to determine which one it is? Should we add another content encoding for this?
Yes, this would eliminate the need for the header. The would be and the definition of would be changed to reference the stream format of a brotli stream with the header prefix. There would be no content-encoding support for a bare brotli dictionary-compressed stream. At that point, maybe renaming them to and for the content encodings would also make things clearer.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: I have a question. If we can use the new and in the header, why do we need to have the "3-byte signature indicating hash and compression type" in the response body?
It's technically not required but it makes it safer and easier to operate on the files outside of the http case. For example, here's some discussion on the brotli issue tracker from 2016 asking for the same: URL
I don't have a problem with doing that in this document, if that is the way people want to go. I personally don't think that a new format is needed because this is a content-encoding, not a media type. But if a media type (and tooling) helps with the deployment of the content-encoding, then maybe that is the right thing to do. Either way, I don't think that you should make that decision (it's a significant one) without having a broader discussion than just this issue.
NAME - could you help me understand this one? (mainly, why wouldn't we be able to apply delta encoding in the absence of an in-body hash?) This will definitely add complexity (e.g. the need to define a new file format that would wrap dictionaries, along with required tooling). It's not currently clear to me what the advantage of this would be. Do we predict the use of this HTTP mechanism outside of an HTTP flow? If so, concrete examples would be helpful.
NAME is the complexity you are concerned about limited to tooling and spec process or do you also see it as being more complex after we are at a good state with tooling? Assuming the brotli and Zstandard libs and cli tools have been updated to add a flag for "embedded dictionary hash" format streams, does one start to look better? For me, the hash being embedded in the stream/file removes fragility from the system. It blocks the decode of the file with the wrong dictionary and enables rebuilding of the metadata that maps dictionaries to compressed files if, for some reason, that metadata got lost (file names truncated, etc). It also feels like it simplifies the serving path a bit, bringing us back to "serve this file" based on the request header vs "serve this file and add this http response header". On the size side of things, I expect it will likely be a wash.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: Delta-compressed responses will be a few bytes smaller because the header (name and value) is larger than the file header (10's of bytes, not huge by any means). In the dynamic resource case where multiple responses will re-use the same dictionary, the header can be compressed away with HPACK/QPACK, making the header case a bit smaller. I don't think it's a big change in complexity/fragility one way or the other but it does feel like there are fewer moving pieces once we get to a place where the tooling is taken care of to have the file contents themselves specify the dictionary they were compressed with. The need for tooling changes would delay adoption a little bit so it's not a free decision but I want to make sure we don't sacrifice future use cases and simplicity for an easier launch.
I'm not sure that I see the tooling process as critical, relative to the robustness. And performance will be a wash (though I tend to view the first interaction as more important than repeated interactions). For tooling, if this content is produced programmatically, then there should be no issue. Integrated tooling can do anything. If content is produced and stored in files, then seems fine. That's a one-way operation generally. I don't see the definition of new media types to be necessary as part of that. I don't see these files being used outside of HTTP, ever. Maybe you could teach the command line decompression tools to recognize the format and complain in a useful way, but that's about the extent of the work I'd undertake there. You could do as NAME suggests as well, which would be even more useful, but given the usage context, that's of pretty narrow applicability.
I'm mostly concerned about official tooling and the latency of getting them to where developers need them to be. Compression dictionaries already require jumping through some , due to latency between brotli releases and adoption by the different package managers. At the same time, if y'all feel strongly about this, this extra initial complexity won't be a deal breaker.
I think there are enough robustness benefits that it is worth some short-term pain that hopefully we will all forget about in a few years.
Code change:
9. <del> The security considerations for Brotli RFC7932 and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </del> <ins> The security considerations for Brotli RFC7932, Shared Brotli SHARED- BROTLI and Zstandard RFC8878 apply to the dictionary-based versions of the respective algorithms. </ins> 9.1.

Related discussion:
text: On the header side of things, how do you all feel with respect to a bare 32-byte hash vs a 35-byte header with a 3-byte signature followed by the hash (or a 3-byte signature followed by a 1-byte header size followed by the hash to allow for changes as well as 4-byte alignment)? It's possible I'm mentally stuck in the old days of sniffing content but since the hash can literally be any value, including accidentally looking like something else, I like the explicit nature of a at the beginning of the stream. It essentially becomes (or whatever the signature is) so it doesn't add significantly to the complexity but it does add 3 bytes, potentially unnecessarily.
+1 to adding a 3 byte magic signature if that's the route we're taking.
I'm ambivalent on the added signature, so I'll defer to others. I can see how that might help a command-line tool more easily distinguish between this and a genuine brotli-/zstd- compressed file and do the right thing with it. On the other hand, it's 3 more bytes and - if the formats themselves have a magic sequence - the same tools could equally skip 32 bytes and check for their magic there.
I think I am having the same response as you all in the opposite direction, where to me it feels preferable to make it HTTP's problem so that my layer doesn't have to deal with additional complexity. :smiley: But if I overcome that bias and accept that it would be nice to avoid an additional HTTP header, here's what I think: If we used a Zstd skippable frame, that would change the stream overhead to 8 bytes (4 byte magic + 4 byte length) + 32 byte hash. But it would mean that existing binaries would be able to decode the frames and avoid ecosystem fragmentation. That's a lot more attractive to me than a new format along the lines of which (1) existing tools won't understand and (2) isn't a format I'd feel comfortable auto-detecting by default in the CLI because isn't all that unlikely a beginning of a string in the way that is (non-ASCII, invalid UTF-8). And I've thought about it more and I'm actually not concerned about colliding a skippable frame type with someone else's existing use case.