jiebi/CodeConvo huggingface dataset:
Code change:
the adversary can control the dictionary, the adversary can learn information about the compressed data. <del> 6.3. </del> <ins> 8.3. </ins> If any of the mitigations do not pass, the client MUST drop the response and return an error. <del> 6.3.1. </del> <ins> 8.3.1. </ins> To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (match) is guaranteed to be for the same origin that the dictionary is served from. <del> 6.3.2. </del> <ins> 8.3.2. </ins> For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user

Related discussion:
text: The alternative floated where 's window size is derived from the size of the dictionary is much more attractive to me. (That is, cap the window size log at , where the spec picks an somewhere in the range .) In such a scheme, where the window size is deterministically derived from the size of the dictionary--a resource the client must already have access to--the client is free to decline to advertise support for for dictionaries which would require the use of a window larger than the client is willing to support. I.e., in resource-constrained environments, Chrome could avoid having to allocate more than 8 MB windows for by not advertising support for on requests where the resolved available dictionary is larger than . Put another way, given that the client will have the tools to make these choices for themselves, I would rather give the client discretion here rather than just bake a flat restriction into the spec.
NAME would you be supportive of updating the zstd command-line tooling to add an option that would set the window to the larger of 8MB or the dictionary size (i.e. match the content-encoding behavior)? My main concern is that it is easy to document and get people to use a fixed command-line for a 8MB window but if it involves a more complicated algorithm then it is likely to break in subtle ways that they may not notice until the resources are bigger than the dictionary (and are bigger than 8MB).
NAME yeah, that would be pretty straightforward. Happy to add that.
If we end up that bundles the dictionary hash with the compressed stream we can make sure the tooling for that file format handles the window size stuff for zstandard automatically (my main concern is around unexpected breakage from different server and client window requirements that only trigger when resources grow). The language will be a bit complex but will probably have to look something like
If I may chime in for some tactical consideration : In an update scenario, it's common for a new resource to be slightly larger than the old one it replaces. The expectation is that we generally tend to add stuff over time. A formulation which caps the window size to the size of the dictionary would introduce problems for all these update cases, which are expected to be the majority of update scenarios. On the other hand, we don't want this increase in size to be completely uncapped, in order to keep control of memory size.
Code change:
the adversary can control the dictionary, the adversary can learn information about the compressed data. <del> 6.3. </del> <ins> 8.3. </ins> If any of the mitigations do not pass, the client MUST drop the response and return an error. <del> 6.3.1. </del> <ins> 8.3.1. </ins> To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (match) is guaranteed to be for the same origin that the dictionary is served from. <del> 6.3.2. </del> <ins> 8.3.2. </ins> For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user

Related discussion:
text: And if the new resource is really much larger than the old one, the benefits of delta compression are dubious anyway. So the suggestion is to authorize a window size which is slightly larger than the external dictionary size. As to how much larger exactly, a suggestion here is by +25% (x1.25). I would expect a large majority of update scenarios to fit into this picture, with the new resource being slightly larger than the old one by a few %. For the less common cases where the new ressource is much larger than the old one, Zstandard could still employ the dictionary but only for the beginning of decompression. The compression process will have to select a window size which remains under the specified limit (x1.25 external dictionary size). As a technical insight, note that byte-accurate window size is only a thing when the window is the entire decompressed content. Otherwise, the window size is encoded in a condensed format, which can only represent increments by 1/8th of a power of 2. This is not a problem to determine this "largest window size under the specified limit", but due to limited accuracy, you can already guess what would be the problem is we were to keep the formulation "not more than the larger of 8 MB and the size of the dictionary". Imagine that the external dictionary size is bytes, and the new resource to delta-compress is , aka 1-byte larger than the dictionary. With the initial formulation, since the window size is limited to the size of the dictionary, the compressor cannot use as a window size, because it's 1 byte larger than the limit. So, it will have to find the next best window size under the limit. Due to granularity limitations, this next best window size is 8 MB. So delta compression would proceed fine for the first 8 MB, but for the last 1 MB, data would have to be compressed without the benefit of the dictionary. With the newly proposed limitation to "x1.25 the dictionary size", delta compression would remain active in above scenario during the entire decompression process. And it would remain fully active if the new resource to decompress is 10 MB, 11 MB or 11.24 MB.
Code change:
the adversary can control the dictionary, the adversary can learn information about the compressed data. <del> 6.3. </del> <ins> 8.3. </ins> If any of the mitigations do not pass, the client MUST drop the response and return an error. <del> 6.3.1. </del> <ins> 8.3.1. </ins> To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (match) is guaranteed to be for the same origin that the dictionary is served from. <del> 6.3.2. </del> <ins> 8.3.2. </ins> For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user

Related discussion:
text: But if the new resource is 11.25 MB or larger, then the compressor will automatically limit the window size to 11 MB (since it's the largest representable window size under the limit, which, in this particular instance, is ), achieving the wanted memory size control.
Since neither the server not the client may know what the resource size will be when compression starts (say, in a streaming HTML case), won't the 1/8th power of 2 granularity apply even for cases where the resource is the same size (or sometimes smaller) than the dictionary? For resources slightly larger than the old one, the dictionary would still be used for the majority of the resource but the last bit would fall back to regular window-based compression. I agree that having the dictionary available for the whole resource would provide better compression but I'm not sure where to draw the line. Something like a 25% fudge factor can be quite a bit when you start talking about 50 MB+ resources that might be close to where the client wants to draw the line anyway. They may have allowed the dictionary at 50 MB but not allow it's use at all if it needed 62 MB available for a window. A similar case could be made for the non-update case where you have a custom-built dictionary but the resource being compressed exceeds 8 MB. Only the first 8 MB of the resource would be able to use the dictionary. Hopefully at that point the compression window would have most of the stuff that was in the dictionary but the dictionary becoming unavailable is something we should expect for some cases with ZStandard, even if we use a fudge factor. Is there a way to convey the 1/8th power of two granularity on the window size or is it something that the ZStandard libraries will take care of automatically for users (i.e. setting the window size to 11 MB when the caller sets it to 11.24 MB)? I'm wondering how much of this implementors are going to have to deal with directly.
It depends if the compression side (server) is informed upfront of the size of the payload to compress. If not, then indeed you are right, the compressor will have to set a Window Size, and the granularity rule applies. So for example, let's assume the payload to delta-compress is 50 MB, and the reference dictionary is also 50 MB.
Code change:
the adversary can control the dictionary, the adversary can learn information about the compressed data. <del> 6.3. </del> <ins> 8.3. </ins> If any of the mitigations do not pass, the client MUST drop the response and return an error. <del> 6.3.1. </del> <ins> 8.3.1. </ins> To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (match) is guaranteed to be for the same origin that the dictionary is served from. <del> 6.3.2. </del> <ins> 8.3.2. </ins> For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user

Related discussion:
text: Let's then assume it implies a window size limit of 50 MB (dictionary size) and the compressor is not informed of the size of the payload. In which case, the compressor will have to set a Window Size, and it will have to settle for the largest representable window size under the limit, which happens to be 48 MB. Therefore, the dictionary won't be used during the compression of the last 2 MB of the payload. With the proposed +25% tolerance rule, the limit would become 62.5 MB, and the largest representable window size under the limit is 60 MB. Now the dictionary remains active during the entire compression of the 50 MB payload. When the data to decompress ends up being smaller than the window size, it's obviously not a problem for the decompression process: the final decompressed size is still properly reported at the end. It's just that the intermediate buffer allocation will be larger than it could have been. Note that uses for the allocation of buffers, so the address space is just reserved, but the physical memory may never be used if it's not needed (i.e. if there is no write action on the later segment part). +25% is more like a suggestion, this can certainly be discussed if some other limit feels better. I liked that it's a (binary) clean factor, and it's larger than , which covers the scenario above, and gives a little bit of breathing room if the newer payload is a bit larger than the older reference one. I see this scenario as being very distinct from delta-compression. For delta-compression, we expect large segments to be identical between the reference and the payload to compress, so we want this ability to access the reference during the entire compression of the payload. The price to pay is that the reference is very specific, and is not designed to offer any serious benefit to any other payload. Custom dictionaries are different: they are supposed to carry the most relevant "commonalities" detected across a set of samples. Presuming a large enough sample set, the common components are not expected to be very large. In such a scenario, the custom dictionary is expected to be (relatively) lightweight (compared to delta-compression), likely in the <= 100 KB range.
Code change:
the adversary can control the dictionary, the adversary can learn information about the compressed data. <del> 6.3. </del> <ins> 8.3. </ins> If any of the mitigations do not pass, the client MUST drop the response and return an error. <del> 6.3.1. </del> <ins> 8.3.1. </ins> To make sure that a dictionary can only impact content from the same origin where the dictionary was served, the URL Pattern used for matching a dictionary to requests (match) is guaranteed to be for the same origin that the dictionary is served from. <del> 6.3.2. </del> <ins> 8.3.2. </ins> For clients, like web browsers, that provide additional protection against the readability of the payload of a response and against user

Related discussion:
text: Consequently, by the time the payload to compress reaches 1 MB, it should already contain all references it needs to compress the rest of the file. I would not expect a large impact of the custom dictionary beyond that point, let alone beyond 8 MB. The plan would be for the library to do this adaptation automatically. The user would not have to know anything about it, though it's better if the user understands that the library selects the closest window size which is <= to the limit, so that they are not surprised if, upon inspection, the final window size is not byte-identical to the set limit. A good API documentation will likely help to reach this objective If by implementors you mean "server side programmers", they would have to know what's the implicit window size limit when a client requests a format (essentially the rule you are currently defining). Then they would set this limit as a compression parameter, and the library will take care of the adaptation into window size.
Yes. (Keeping in mind of course that most traffic in question will presumably be under the 8 MB floor and so none of this complexity will apply.) Right now the zstd library has limited controls about setting detailed window constraints. We'll need to add a new option to give users that fine-grained control. We can do that with an eye towards user simplicity. Frankly, I'm also very open to also adding a flag (and equivalent compression parameter in the library) that would just look at the size of the dictionary (which zstd necessarily has access to) and would configure the window accordingly. In that case, users wouldn't have to do any math at all.
OK, I chaged the structure and language so that they are no longer streams of bytes compressed with a given algorithm but are now specifically and . The references for those named formats have the window restrictions (which for Zstandard includes the 1.25x dictionary requirement). Currently the byte stream itself is still a native Brotli/Zstandard stream, just with specific settings but this also makes it cleaner if we decide to . PR with the update is here: URL
lgtm with nits, thank you!LGTM!
Code change:
The value of the header is encoded as described in encoding. The "Client-Cert" header field is only for use in HTTP requests and <del> MUST NOT be used in HTTP responses.  It is a single HTTP header field value as defined in Section 3.2 of RFC7230, which MUST NOT have a </del> <ins> MUST NOT be used in HTTP responses.  It is a singleton header field value as defined in Section 5.5 of RFC9110, which MUST NOT have a </ins> list of values or occur multiple times in a request. 2.3. In the context of a TLS terminating reverse proxy deployment, the <del> proxy MAY make the certificate chain used for validation of the end- entity certificate available to the backend application with the Client-Cert-Chain HTTP header field.  This field contains certificates used by the proxy to validate the certificate used by the client in the TLS handshake.  These certificates might or might not have been provided by the client during the TLS handshake. </del> <ins> proxy MAY make the certificate chain available to the backend application with the Client-Cert-Chain HTTP header field. </ins> Client-Cert-Chain is a List Structured Header RFC8941.  Each item in the list MUST be a Byte Sequence (RFC8941) encoded as described in <del> encoding. </del> <ins> encoding.  The order is the same as the ordering in TLS (such as described in TLS). </ins> The header's ABNF is:

Related discussion:
text: Martin kindly suggested that I was making this too complicated and to just make the change (paraphrasing). I've tried to do that with changes in PR .
This works well enough on the critical points. Some points about readability though.
Code change:
Each leaf in a CCF ledger carries the following components: <del> The "internal_hash" and "internal_data" byte strings are internal to the CCF implementation.  Similarly, the auxiliary tree entries are internal to CCF.  They are opaque to receipt Verifiers, but they commit the TS to the whole tree contents and may be used for additional, CCF-specific auditing. </del> <ins> The "internal-transaction-hash" and "internal-evidence" byte strings are internal to the CCF implementation.  They can be safely ignored by receipt Verifiers, but they commit the TS to the whole tree contents and may be used for additional, CCF-specific auditing. </ins> 3.

Related discussion:
text: NAME NAME I have read through URL, and made some edits here as discussed to correct the leaf definition. This is deliberately quite light on details, but I am obviously very happy to expand wherever necessary.
NAME I had not seen the updates made in URL, I have attempted to merge them in here.
NAME if/when you have a chance, I would be grateful for a review and some feedback on how to take this forward :)
NAME I have tried to address your feedback, I hope I did not miss anything.
NAME may I ask you to check if the changes correctly addressed your feedback, and what the next steps are?
NAME thank you for the discussion yesterday, I have and added requested assignments for the labels. Please let me know if I need to do anything else before this can be merged.
:+1: let's pull this in and prepare for WG adoption call
Code change:
The format of SUBSCRIBE REQUEST is as follows: <del> Full Track Name: Identifies the track as defined in (track-name). </del> <ins> Track Namespace: Identifies the namespace of the track as defined in (track-name). Track Name: Identifies the track name as defined in (track-name). </ins> Track Request Parameters: As defined in track-req-params.

Related discussion:
text: by explicitly specifying the namespace. Also consistently changes to the (b) notation to indicate a variable length sequence of bytes.
This PR got lost when merging 289.
Assume a setup with two publishers, a single relay and one subscriber. P1 ANNOUNCE's track namespace fooba P2 ANNOUNCE's track namespace foo S1 sends a SUBSCRIBE REQUEST for Full Track Name How does the relay know whether is Track Namespace: and Track Name: , or Track Namespace: and Track Name , and hence which publisher to route the subscribe to? I see a few possibilities: 1) The relay should not allow this scenario (via namespace construction and/or authorization). If so the draft should include text explaining that this is disallowed (normatively?) or "bad things(tm)" happen. 2) Change SUBSCRIBE REQUEST to be able to disambiguate the Track Namespace from the Track Name. This could be done by sending name and namespace in separate message fields, or having stricter rules for the structure of Full Track Name such that a relay can extract the constituent parts. 3) Explicitly state that Full Track Names are matched to namespaces via "longest prefix match".
Option 3 looks interesting. P1 can use a longer track namespace to steer the sub request from P2. Like a traffic engeering for content.
I like option 2. I don't agree with treating track namespace as merely a string prefix. Even if you go with option 3, there's still a race condition where a late ANNOUNCE will cause a SUBSCRIBE to an unintended track. I want to give the track namespace more properties so it's treated more like a broadcast identifier. A few folks are headed down this path without realizing it, such as proposing that there's a single catalog per track namespace. I don't see how that would even work if ANNOUNCE is used purely for routing like a BGP announcement.
NAME : are you saying you believe that the relay resolves the described ambiguity by using 'longest prefix' matched (eg: option 3)? Another option I hadn't considered based on this comment is that the relay would send OBJECTs from BOTH publishers since they both match the Full Track Name. This will break in the current design, since each publisher has their own Object Sequence and Group Sequence and when merged the consumer will be left with garbage.
Code change:
The format of SUBSCRIBE REQUEST is as follows: <del> Full Track Name: Identifies the track as defined in (track-name). </del> <ins> Track Namespace: Identifies the namespace of the track as defined in (track-name). Track Name: Identifies the track name as defined in (track-name). </ins> Track Request Parameters: As defined in track-req-params.

Related discussion:
text: It also violates the logic described that says a Full Track Name can be used as a cache key.
Yeah that seems quite problematic NAME Let's say a relay have two connected publishers who send IN ORDER: A subscriber connects and issues: What happens next? The relay forwards the SUBSCRIBE to publisher 1. The relay returns SUBSCRIBE ERROR notfound The relay forwards the SUBSCRIBE to publisher 2. The relay forwards the SUBSCRIBE to publisher 1&2. These are in the order of Alan's options. I added option 4 based on Suhas' comment. Would occur if the relay rejects publisher2's ANNOUNCE because of thr shared prefix. Would occur because the track namespace is an ID, not a prefix, and must fully match an ANNOUNCE. Would occur because publisher2 has the longer path prefix. Would occur because both publishers could_ produce the requested track. And to throw a wrench in things, a third publisher joins.
There's also the case where the relay replies to the subscribe requests on behalf of the publisher(s), so it doesn't have to make the routing decision at subscription time. But it does need to decide where to forward OBJECTs when they arrive.
Implementing this now so I have some more thoughts. We should explicitly support UNANNOUNCE and UNSUBSCRIBE, which implicitly occur when the connection is dropped. On ANNOUNCE, the relay MUST ensure that the namespace not a subset/superset of an existing ANNOUNCE. On SUBSCRIBE, the relay looks up the track based on the prefix and appends itself to a list of subscribers. If not found, it looks up the namespace and creates a track entry if valid. On UNSUBSCRIBE, the relay looks up the track based on the full name and removes itself from a list of subscribers. On UNANNOUNCE, the relay deletes any namespaces and tracks matching the prefix. A radix tree is used for both so the relay can iterate through any values that match the given prefix. This is a global radix tree, similar to how QUIC connections are identified when variable length connection IDs are supported. On ANNOUNCE, the relay inserts the upstream into the upstreams hash map and errors on a duplicate. On SUBSCRIBE, the relay looks up the namespace and then the track, creating an entry if not found.