jiebi/CodeConvo huggingface dataset:
Addressing issue and
This fixes issue and issue
Please add "and is still valid" - see new proposed langauge enable the consumer to verify that software originated from a 'duly authorized signing party' on behalf of the Supplier, and is still valid
+1 to this change
Closing as been addressed now!
I have no concerns with the revised language for 3.1. One minor nit on 3.2 to be consistent with the use case title REPLACE: A consumer of released software component wants: WITH A consumer of released software product wants:
LGTM
This issue has been resolved in the above PR.
LGTM
Good point.
I guess all the options are ugly. Should we leave the text wehre ut was and just apply the proposed change in URL ?
Does better belong in -messaging?
Into 6.2?
Seems like the best place, yeah.
NAME : URL
But I don't know what this accomplishes given the paragraph after this one also refers to Transfer-Encoding.This makes sense on its own (with Martin's adjustment). Still not thrilled about having transfer-codings in semantics.Thanks. Fairly simple fix here. I've suggested a tweak for Roy's concern (which I can't make a suggestion before because of GitHub limitations.
references RFC6515 and 6194 for deprecating md5 and sha1 Since there are RFCs, we could even skip explaining that those algorithms are vulnerable to collisions.
cc: NAME
NAME feel free to PR for the rest of the algorithms :P
In s12.2, should refer (informatively) to and for IETF references that deprecate the use of md5 and sha1, respectively.
Thanks for the pointers, sgtm
You beat me to the PR!
After looking harder, I found that RFC 9110 that a proxy "is usually identified by an "http" or "https" URI". I'm not sure that's 100% accurate, but it conveniently resolves the wording problem, and who am I to argue with RFC 9110?
At the moment, section 1.2 (Problems) start with the following sentence: I totally get what it tries to say, but it is confusing if not incorrect. states that a CONNECT request is handled by a "proxy" and the target specified by the "host:port" tuple is the "origin." Therefore, HTTP CONNECT proxies are not identified by an origin. Rather, they are identified by the user's proxy settings. I think we'd better update the problem statement, and might also consider pointing out that the definition of "origin" will be different between CONNECT and connect-tcp (or use different terms to identify the CONNECT target and the entity that creates a tunnel in order to avoid confusion, maybe?).
PS. Practical example showcasing the side effect of the change to the definition is that the way to authenticate becomes different; when a client tries to establish a CONNECT tunnel with authentication, it uses "proxy-authorization" header, whereas when a client tries to establish a connect-tcp tunnel, it uses "authorization." I think it's worth noting.
I'm realizing that this spec has the opportunity to fix the origin issues present in regular CONNECT. While we're there it'd be great to clarify how connect-tcp interacts with Alt-Svc because it's unclear for CONNECT. I sent some more about why it's unclear for CONNECT.
Thanks for the discussion!
Removes a duplicated section from the introduction
Feedback from Brian Campbell: Section 2 repeats itself with the text "A client in trust domain A that needs to access a resource server in trust domain B requests an authorization grant from the authorization server for trust domain A via a token exchange. The client in trust domain A presents the received grant as an assertion to the authorization server in domain B in order to obtain an access token for the protected resource in domain B. The client in domain A may be a resource server, or it may be the authorization server itself." occurring twice.
Me confused. Which PR is relevant now? Both?
The other one is an evolution of this one.
We use the phrases 'selected representation' and 'selected response' in Semantics and Caching (respectively) to refer to two completely different concepts. Can we do better?
No
I think the two concepts match pretty well (and are used correctly in the four sentences). However, I also think that "chosen response" would be okay given it is only used in four places. I really don't think it needs a separate definition, since it means what the two words mean. So, maybe.
It was actually used in a few more places as the response selected. I think I found all of them.
See .
This eliminates the response header and moves the hash of the dictionary into the payload of the response. Specifically, it adds 36-byte header in front of the compressed stream which is a 4 or 8-byte signature followed by the 32-byte SHA-256 hash digest. The signature is different for the two different compressions: Dictionary-Compressed Brotli: (0xFF followed by ) Dictionary-Compressed Zstandard: (32-byte skippable Zstandard frame) This also changes the content-encoding names for the different schemes to make it clear that they are a new format and not a raw brotli or Zstandard stream ( and ). From the cli, the stream is equivalent to: or
FWIW, it doesn't look like it would be particularly difficult to use a skippable frame instead of a custom header in the Zstandard case if that's the way we want to go. It's 4-bytes bigger (for the frame size) so it will essentially have an 8-byte "magic number" (since the size will always be the same). It won't help with the creation of the files but if there's benefit to have the existing tooling be able to decode them (without verifying the hash because it will just skip it) we can go that route. On the client side, it feels like it will be cleaner on the implementation to use the same flow for verifying the dictionary hash and magic number for both encodings without having to deal with different sized magic numbers (and always buffer and remove the first 36 bytes) but it's not a significant difference.
Still LGTM
NAME could you please take a look and make sure I got the byte sequence for the skippable frame correct? I tested with the zstd cli and it was able to decode the file with the header without issue when created like this:
Thanks. Updated and added an example with the zstd cli.
This is a discussion we've had several times when defining content codings, but it seems like there is never really a single answer. Should the content coding be self-describing, or can it rely on metadata in fields? The compression dictionary work uses header fields to identify which compression dictionary is in use. Originally, the client would indicate a dictionary and the server would indicate use of that dictionary by choosing the content coding.
This made interpreting the body of the response quite challenging in that you needed to have a request in order to make sense of it. More recently, the specification has changed to having the client list available dictionaries, with the server echoing the one it chooses. Both use header fields. There is a third option, which is to embed the dictionary identification (which is a hash of the dictionary) ahead of the compressed content. This has some real advantages: Requests could use delta encoding. There is - of course - a real question about availability of the dictionary, but we have that problem with encrypted content codings and keys. That is a problem we solve for responses with a field. It might be solved using the same technique, combined with the client-initiated content coding advice. Or applications can use their own systems. A field costs a lot more bits. This would not benefit from header compression, so it might be a net loss in some cases, but in many cases it would be a distinct win because 32 bytes of hash in the body of a request is far less than a field name and field value containing a base64 encoding of those same 32 bytes. It also comes with disadvantages: Header compression could remove a lot of the cost in bytes. It might be ever-so-slightly more complex for encoding and decoding to have to split the first 32 bytes off the body of a message.
On the encoding side, the main downside is that the cli tooling for brotli and Zstandard don't currently do the embedding so tooling would have to be added to prepend the hash to the files in a way that isn't standard for either (yet anyway) and for manually decompressing the files. Zstandard has identifiers for the dictionaries when using non-raw dictionaries but both assume that raw dictionaries will be negotiated out of band. Technically it would be a pretty trivial modification for clients and servers that are doing the work, I'm just a bit concerned about the developer experience changes (and whatever needs to be done to get both brotli and Zstandard to understand what amounts to new file formats).
I opened issues with brotli and ZStandard to see if they would consider adding it to the format of their respective file formats.
If it's an optional metadata tag that is backwards compatible with existing encoders and decoders I could see it providing quite a bit of value, even in the non-HTTP case of dictionary compression.
There was some discussion in the ZStandard repo about possibly reserving one of the skippable frame magic numbers for embedding a dictionary ID but there's some level of risk for collision with people who may be using those frames for watermarking or other application-specific use cases. As best as I can tell, the brotli stream format doesn't have a similar frame capability for metadata or application-specific data. We could create a container format that held the dictionary ID and stream (header basically, not unlike zip vs deflate) but that feels like a fairly large effort and the tooling would have to catch up to make it easy for developers to work with. At this point I'm hesitant to recommend adding anything to the payload itself that the existing brotli and zstd tooling can't process. Being able to create, fetch and test the raw files is quite useful for the developer workflow and for debugging deployments. Would it make sense to allow for future encoding formats to include a dictionary ID in the file itself and make the header optional in those cases (and make the embedded ID authoritative)? I'm not sure if that would make sense in this draft since this one is limited to the 2 existing encodings and it would be addressed in a new draft when the new encodings are created or if it makes sense to allow for it here without requiring that and explicitly require embedded ID's.
I don't think that you want to change the zstd or brotli format, only the content coding. That is, something like this: This does partly work against the idea that you might have a bunch of files that contain compressed versions of content. You can't just say . And you can't just decompress without some processing. What you get in exchange is a message payload with the advantages I described.
I do think we need to define a file format for it if we go down this path to ease adoption and it should probably have a magic signature at the beginning. Maybe something like gzip is to deflate but with a simple 3-byte signature followed by the hash followed by the stream data.
Assuming we create a cli tool to do all of the work for compressing/decompressing them, I'll ping several of the current origin trial participants to see how it will fit into their workflow. Something like: Dictionary-compressed file/stream format with a 3-byte signature indicating hash and compression type, followed by dictionary hash, followed by raw compressed stream. Something like for Dictionary-Compressed Brotli with Sha-256 hash and for Dictionary-Compressed Zstandard with Sha-256 hash. Create a cli script (cross-platform, available in npm, bun, apt, etc) that creates the files and (optionally) calls out to brotli and zstd to do the compression. We could include any special ZStandard compression window handling in the format spec as well as the cli tool. I'm assuming something like this would be better off as it's own draft that this references or do you think it makes sense to define it here? I agree there are significant benefits to having the hash paired directly with the resource. I just want to be careful to make sure whatever we do fits in well with the developer workflow as well.
Adding the header would add some work to typical CI workflow. At least in my case, the diff file stream is created using the cli using the options. Adding the header would require creating a new file with the header and add the generated stream. If this can be done in a published script it would simplify the logic (FYI - URL currently does yet not support brotli bindings, see URL) but not a requirement IMO. Do I understand correctly that with this idea implemented the header would not be necessary anymore? That would simplify some CDN configurations which currently needs to add it. Related to this - what in this case would be header value? current spec is it should be which I understand as brotli dictionary stream. If we add the header, how would a client be able to determine which one it is? Should we add another content encoding for this?
Yes, this would eliminate the need for the header. The would be and the definition of would be changed to reference the stream format of a brotli stream with the header prefix. There would be no content-encoding support for a bare brotli dictionary-compressed stream. At that point, maybe renaming them to and for the content encodings would also make things clearer.
I have a question. If we can use the new and in the header, why do we need to have the "3-byte signature indicating hash and compression type" in the response body?
It's technically not required but it makes it safer and easier to operate on the files outside of the http case. For example, here's some discussion on the brotli issue tracker from 2016 asking for the same: URL
I don't have a problem with doing that in this document, if that is the way people want to go. I personally don't think that a new format is needed because this is a content-encoding, not a media type. But if a media type (and tooling) helps with the deployment of the content-encoding, then maybe that is the right thing to do. Either way, I don't think that you should make that decision (it's a significant one) without having a broader discussion than just this issue.
NAME - could you help me understand this one? (mainly, why wouldn't we be able to apply delta encoding in the absence of an in-body hash?) This will definitely add complexity (e.g. the need to define a new file format that would wrap dictionaries, along with required tooling). It's not currently clear to me what the advantage of this would be. Do we predict the use of this HTTP mechanism outside of an HTTP flow? If so, concrete examples would be helpful.
NAME is the complexity you are concerned about limited to tooling and spec process or do you also see it as being more complex after we are at a good state with tooling? Assuming the brotli and Zstandard libs and cli tools have been updated to add a flag for "embedded dictionary hash" format streams, does one start to look better? For me, the hash being embedded in the stream/file removes fragility from the system. It blocks the decode of the file with the wrong dictionary and enables rebuilding of the metadata that maps dictionaries to compressed files if, for some reason, that metadata got lost (file names truncated, etc). It also feels like it simplifies the serving path a bit, bringing us back to "serve this file" based on the request header vs "serve this file and add this http response header". On the size side of things, I expect it will likely be a wash.
Delta-compressed responses will be a few bytes smaller because the header (name and value) is larger than the file header (10's of bytes, not huge by any means). In the dynamic resource case where multiple responses will re-use the same dictionary, the header can be compressed away with HPACK/QPACK, making the header case a bit smaller. I don't think it's a big change in complexity/fragility one way or the other but it does feel like there are fewer moving pieces once we get to a place where the tooling is taken care of to have the file contents themselves specify the dictionary they were compressed with. The need for tooling changes would delay adoption a little bit so it's not a free decision but I want to make sure we don't sacrifice future use cases and simplicity for an easier launch.
I'm not sure that I see the tooling process as critical, relative to the robustness. And performance will be a wash (though I tend to view the first interaction as more important than repeated interactions). For tooling, if this content is produced programmatically, then there should be no issue. Integrated tooling can do anything. If content is produced and stored in files, then seems fine. That's a one-way operation generally. I don't see the definition of new media types to be necessary as part of that. I don't see these files being used outside of HTTP, ever. Maybe you could teach the command line decompression tools to recognize the format and complain in a useful way, but that's about the extent of the work I'd undertake there. You could do as NAME suggests as well, which would be even more useful, but given the usage context, that's of pretty narrow applicability.
I'm mostly concerned about official tooling and the latency of getting them to where developers need them to be. Compression dictionaries already require jumping through some , due to latency between brotli releases and adoption by the different package managers. At the same time, if y'all feel strongly about this, this extra initial complexity won't be a deal breaker.
I think there are enough robustness benefits that it is worth some short-term pain that hopefully we will all forget about in a few years.
On the header side of things, how do you all feel with respect to a bare 32-byte hash vs a 35-byte header with a 3-byte signature followed by the hash (or a 3-byte signature followed by a 1-byte header size followed by the hash to allow for changes as well as 4-byte alignment)? It's possible I'm mentally stuck in the old days of sniffing content but since the hash can literally be any value, including accidentally looking like something else, I like the explicit nature of a at the beginning of the stream. It essentially becomes (or whatever the signature is) so it doesn't add significantly to the complexity but it does add 3 bytes, potentially unnecessarily.
+1 to adding a 3 byte magic signature if that's the route we're taking.
I'm ambivalent on the added signature, so I'll defer to others. I can see how that might help a command-line tool more easily distinguish between this and a genuine brotli-/zstd- compressed file and do the right thing with it. On the other hand, it's 3 more bytes and - if the formats themselves have a magic sequence - the same tools could equally skip 32 bytes and check for their magic there.
I think I am having the same response as you all in the opposite direction, where to me it feels preferable to make it HTTP's problem so that my layer doesn't have to deal with additional complexity. :smiley: But if I overcome that bias and accept that it would be nice to avoid an additional HTTP header, here's what I think: If we used a Zstd skippable frame, that would change the stream overhead to 8 bytes (4 byte magic + 4 byte length) + 32 byte hash. But it would mean that existing binaries would be able to decode the frames and avoid ecosystem fragmentation. That's a lot more attractive to me than a new format along the lines of which (1) existing tools won't understand and (2) isn't a format I'd feel comfortable auto-detecting by default in the CLI because isn't all that unlikely a beginning of a string in the way that is (non-ASCII, invalid UTF-8). And I've thought about it more and I'm actually not concerned about colliding a skippable frame type with someone else's existing use case.
It would be more of a problem if we were trying to spec a universal solution, but if we scope this to just this content-encoding, then we're free to reserve whatever code points we want and attach whatever semantics we want.
Thanks. I guess the main question I have is if there would be benefits to Zstandard itself for the files to be self-contained with the identification of the dictionary that they were encoded with or the possibility of mismatching dictionaries on compression and decompression (or being able to find the matching dictionary given just the compressed file) are issues that are HTTP-specific. I'm fine with specifying that the encoded files carry the dictionary hash (before the compressed stream data) and having different ways for Zstandard and Brotli to do that actual embedding. That said, the tooling gets more complicated on the encode and decode side to generate the frame and insert it in the correct place in the compressed file and at decompression time, that makes the client more format-aware, having to parse more of the stream to extract the dictionary and then re-send the full stream through the decoder (at least until the decoder library becomes aware of embedded dictionary hash).
NAME Is this really what you want in this case? The decoder needs to know where to find the dictionary, so wouldn't you want this to be a breaking change to the format such that a decoder that has a dictionary is fine and a decoder that doesn't knows to go get one. ... And - importantly - an older decoder will abort. (I confess that I don't know what the zstd frame extension model is and didn't check.)
I don't think we're contemplating a model where Zstd can ingest a frame and figure out on its own where to find the dictionary it needs and then load it and use it. I expect that the enclosing application will parse this header, get the hash, and then find the dictionary and provide it to Zstd. The advantage to using the skippable frame is that then you can provide the whole input to Zstd (including existing versions) and it will work rather than have to pull the header off.
One thing I realized now - by assuming that the hash length is 32 bytes, we're assuming the hash would remain SHA-256 forever. That might be how things play out, but it that we'd need to change hashes at some point.
If we were to do that, having a fixed length hash as part of the format would make things more complex.
Having a fixed-length hash (or fixed hash) as part of a content coding is perfectly fine. If there is a need to update hashes, it is easy to define a new content coding.
NAME the doesn't use skippable frames and uses the same custom header for Brotli and Zstandard (with different magic numbers). I can switch to using a skippable frame instead (which effectively just becomes an 8-byte magic number since the frame length is always the same) but I'm wondering if it makes sense and is worth adding 4 bytes. It won't help in creating the files so it's just for decode time and the main benefit you get is that you can use the existing zstd cli and libraries to decode the stream without skipping the header but those also won't verify the dictionary hash, they will just skip over it. That might not be a problem but part of the decode process will be to fail the request if the hashes don't match.
In talking to the Brotli team, it looks like Brotli already and validates it during decode to select the dictionary to use. It uses a "256-bit Highwayhash checksum" so we can't use the hash to lookup a dictionary indexed by SHA-256 but we can use it to guarantee the decompression doesn't use a different dictionary (and the existing libraries and cli tools already use it). NAME when you were concerned about the , was it for both lookup and validation or just validation? I'm just wondering if we can use the existing brotli streams as they already exist or if we should still add a header to support locating the dictionary by SHA-256 hash.
There are a few places in HTTP where the interpretation of a response depends on something in the request. Everyone one of those turns out to be awful for generic processing of responses. That's my main reasoning, so I'd say both: lookup first, then validation. That is, my expectation is not that the client has a singular candidate dictionary, or that it commits to one in particular. So in the original design, when you had the client pick one, that didn't seem like a great idea to me. For validation, Highwayhash (NIH much?)