jiebi/CodeConvo huggingface dataset:
Code change:
When there are multiple dictionaries that match a given request URL, the client MUST pick a single dictionary using the following rules: <del> 1.  For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination.  1.  Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. 1.  Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </del> <ins> For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination. Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </ins> 2.3.

Related discussion:
text: IIRC, regular (as opposed to large-window) brotli allows distances up to ~56MiB, i.e. larger shared dictionaries would not be useful (== referenced)
Thanks. For the dictionaries to be usable for a broad set of apps (including large wasm bundles), I'm considering spec'ing: Brotli: Regular 16MB window, allowing for delta-compression of resources up to ~50MB. ZStandard: 128MB window, allowing for delta compression of resources up to around that size, depending on where the duplicated bytes are in the updated resource. The main difference with regular content-encoding is on the ZStandard side where the window is 8MB. For platforms that are resource constrained and are not willing to support a 128MB window they would be limited to Brotli (and then decide how large of a dictionary they are willing to keep and use). Another option on the ZStandard side would be to allow for a maximum window that is a multiple of the dictionary size (i.e. 8MB or 2 * dictionary size whichever is larger with a 128MB max). That way the client can have an upper limit that it knows in advance before advertising as being available. That adds complexity to the encoding side though. We have already seen cases where people want to delta-compress resources > 50MB (wasm apps) where they had to use ZStandard instead of Brotli because of the Brotli limits so I don't think a small limit like 8MB for ZStandard would work with this use case.
That massive resources exist is not sufficient justification to build a system that will not be available to some users. I would prefer a far smaller default. If only to encourage the use of smaller resources overall. Defining a separate codepoint for massive windows is possible, say . The scalable option also seems reasonable. Clients can then choose to offer dictionaries when they exceed available resources.
I'm ok using the same 8MB window ( I believe) that we use for for and note that the effectiveness of delta compression for responses > 8MB using may be limited. The scalable option concerns me that people are likely to get it wrong and it won't be obvious and the decompression will start to fail as the dictionary and resource size changes. NAME does that sound about right?
Code change:
When there are multiple dictionaries that match a given request URL, the client MUST pick a single dictionary using the following rules: <del> 1.  For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination.  1.  Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. 1.  Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </del> <ins> For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination. Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </ins> 2.3.

Related discussion:
text: And if the new resource is really much larger than the old one, the benefits of delta compression are dubious anyway. So the suggestion is to authorize a window size which is slightly larger than the external dictionary size. As to how much larger exactly, a suggestion here is by +25% (x1.25). I would expect a large majority of update scenarios to fit into this picture, with the new resource being slightly larger than the old one by a few %. For the less common cases where the new ressource is much larger than the old one, Zstandard could still employ the dictionary but only for the beginning of decompression. The compression process will have to select a window size which remains under the specified limit (x1.25 external dictionary size). As a technical insight, note that byte-accurate window size is only a thing when the window is the entire decompressed content. Otherwise, the window size is encoded in a condensed format, which can only represent increments by 1/8th of a power of 2. This is not a problem to determine this "largest window size under the specified limit", but due to limited accuracy, you can already guess what would be the problem is we were to keep the formulation "not more than the larger of 8 MB and the size of the dictionary". Imagine that the external dictionary size is bytes, and the new resource to delta-compress is , aka 1-byte larger than the dictionary. With the initial formulation, since the window size is limited to the size of the dictionary, the compressor cannot use as a window size, because it's 1 byte larger than the limit. So, it will have to find the next best window size under the limit. Due to granularity limitations, this next best window size is 8 MB. So delta compression would proceed fine for the first 8 MB, but for the last 1 MB, data would have to be compressed without the benefit of the dictionary. With the newly proposed limitation to "x1.25 the dictionary size", delta compression would remain active in above scenario during the entire decompression process. And it would remain fully active if the new resource to decompress is 10 MB, 11 MB or 11.24 MB.
Code change:
When there are multiple dictionaries that match a given request URL, the client MUST pick a single dictionary using the following rules: <del> 1.  For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination.  1.  Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. 1.  Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </del> <ins> For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination. Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </ins> 2.3.

Related discussion:
text: Let's then assume it implies a window size limit of 50 MB (dictionary size) and the compressor is not informed of the size of the payload. In which case, the compressor will have to set a Window Size, and it will have to settle for the largest representable window size under the limit, which happens to be 48 MB. Therefore, the dictionary won't be used during the compression of the last 2 MB of the payload. With the proposed +25% tolerance rule, the limit would become 62.5 MB, and the largest representable window size under the limit is 60 MB. Now the dictionary remains active during the entire compression of the 50 MB payload. When the data to decompress ends up being smaller than the window size, it's obviously not a problem for the decompression process: the final decompressed size is still properly reported at the end. It's just that the intermediate buffer allocation will be larger than it could have been. Note that uses for the allocation of buffers, so the address space is just reserved, but the physical memory may never be used if it's not needed (i.e. if there is no write action on the later segment part). +25% is more like a suggestion, this can certainly be discussed if some other limit feels better. I liked that it's a (binary) clean factor, and it's larger than , which covers the scenario above, and gives a little bit of breathing room if the newer payload is a bit larger than the older reference one. I see this scenario as being very distinct from delta-compression. For delta-compression, we expect large segments to be identical between the reference and the payload to compress, so we want this ability to access the reference during the entire compression of the payload. The price to pay is that the reference is very specific, and is not designed to offer any serious benefit to any other payload. Custom dictionaries are different: they are supposed to carry the most relevant "commonalities" detected across a set of samples. Presuming a large enough sample set, the common components are not expected to be very large. In such a scenario, the custom dictionary is expected to be (relatively) lightweight (compared to delta-compression), likely in the <= 100 KB range.
Code change:
When there are multiple dictionaries that match a given request URL, the client MUST pick a single dictionary using the following rules: <del> 1.  For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination.  1.  Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. 1.  Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </del> <ins> For clients that support request destinations, a dictionary that specifies and matches a "match-dest" takes precedence over a match that does not use a destination. Given equivalent destination precedence, the dictionary with the longest "match" takes precedence. Given equivalent destination and match length precedence, the most recently fetched dictionary takes precedence. </ins> 2.3.

Related discussion:
text: Consequently, by the time the payload to compress reaches 1 MB, it should already contain all references it needs to compress the rest of the file. I would not expect a large impact of the custom dictionary beyond that point, let alone beyond 8 MB. The plan would be for the library to do this adaptation automatically. The user would not have to know anything about it, though it's better if the user understands that the library selects the closest window size which is <= to the limit, so that they are not surprised if, upon inspection, the final window size is not byte-identical to the set limit. A good API documentation will likely help to reach this objective If by implementors you mean "server side programmers", they would have to know what's the implicit window size limit when a client requests a format (essentially the rule you are currently defining). Then they would set this limit as a compression parameter, and the library will take care of the adaptation into window size.
Yes. (Keeping in mind of course that most traffic in question will presumably be under the 8 MB floor and so none of this complexity will apply.) Right now the zstd library has limited controls about setting detailed window constraints. We'll need to add a new option to give users that fine-grained control. We can do that with an eye towards user simplicity. Frankly, I'm also very open to also adding a flag (and equivalent compression parameter in the library) that would just look at the size of the dictionary (which zstd necessarily has access to) and would configure the window accordingly. In that case, users wouldn't have to do any math at all.
OK, I chaged the structure and language so that they are no longer streams of bytes compressed with a given algorithm but are now specifically and . The references for those named formats have the window restrictions (which for Zstandard includes the 1.25x dictionary requirement). Currently the byte stream itself is still a native Brotli/Zstandard stream, just with specific settings but this also makes it cleaner if we decide to . PR with the update is here: URL
lgtm with nits, thank you!LGTM!
Code change:
When a stream begins with "STREAM_HEADER_TRACK", all objects on the stream belong to the track requested in the Subscribe message identified by "Subscribe ID".  All objects on the stream have the <del> "Object Send Order" specified in the stream header. </del> <ins> "Publisher Priority" specified in the stream header. </ins> All Objects received on a stream opened with STREAM_HEADER_TRACK have an "Object Forwarding Preference" = "Track".

Related discussion:
text: There are some obsolete references to Object Send Order, and part of the STREAMHEADERTRACK description seems to have slipped below the header for the following section.
Thanks for fixing these .. LGTM
Code change:
Append the result of running Serializing an Item (ser-item) with (member_value, parameters) to output. <del> If more values remain in inner_list, append a single WS to </del> <ins> If more values remain in inner_list, append a single SP to </ins> output. Append ")" to output.

Related discussion:
text: An alternative to - removes HTAB everywhere, replaces with .
Right now we use OWS, and suggests moving from SP to RWS for inner list separation. Should we allow tabs in SH? Error on them? If we want to allow them, 983 makes sense, as it's more consistent. However, we'll need to test for tabs...
What's the reason to allow tabs? Compatibility with current headers? Do actual tab characters show up in the wild that often?
We had them in there because we were reusing OWS, not purposefully. AFAICT they don't appear in the wild much at all. So +1 for .
LGTMThank you for working on this. LGTM.
Code change:
Track Namespace: Identifies a track's namespace as defined in (track-name). <del> 6.14. </del> <ins> 6.13. </ins> The subscriber sends an "ANNOUNCE_CANCEL" control message to indicate it will stop sending new subscriptions for tracks within the provided

Related discussion:
text: Into SUBSCRIBEDONE. I didn't remove the SUBSCRIBEERROR message in favor of SUBSCRIBE_CLOSED as NAME suggested in this PR. Addresses some comments in , but possibly not all
So my notes from this discussion. CLOSE is confusing with QUIC layer. People seemed to prefer UNSUBSCRIBE over SUBSCRIBE_CLOSE. I love the move aware from RST and FIN as naming.
Propose names from my understanding of discussion call Announce (send by publisher) , AnnouceOK, AnnouceError UnAnnounce (send by publisher) UnAnnounceOK, UnAnnounceError AnnouceEnd ( sent by thing that received Annouce) , AnnouceEndOK, AnnouceEndOK Subscribe (send by subscriber) , SubscribeOK, SubscribeError UnSubscribe (send by subscriber) , SubscribeOK, SubscribeError SubscribeEnd (send by thing that received subscriber), SubscribeOK, SubscribeError Mostly trying to get rid of FIN / RST and have different names for the messages that flow in opposite directions.
I don't like the name of the different messages used to terminate. I'd rather have one message that transitions to the terminal state, with an optional error code and final group/object. IMO the subscription flow: And the ANNOUNCE flow mirrors: The brackets mean a message is optional. The sender is responsible for creating and the receiver is responsible for closing, possibly at the bequest of the sender (like QUIC's STOP_SENDING).
has been merged, adding yet another term (CANCEL), but I figured we'd clean that up with this issue.
Since we are bikeshedding, I prefer ANNOUNCE_RESCIND instead of CANCEL, since an announce is more of a declarative statement than an active request from the sender.
Heh and UNANNOUNCE and UNSUBSCRIBE stick out because they don't share the same prefix.
I like the names being proposed as they help make make is easier to understand. I do want to separate out one issue. The STOP / CLOSE type thing happens way after the initial SUBSCRIBE thus it can end up in a situation where auth has expired or something else has gone wrong this I like the current design that allows an OK or ERROR to be returned for those. I think it is also key for sender of the things stoping the subscribe to know that the other side has acted on that so we need the OK.
Code change:
Track Namespace: Identifies a track's namespace as defined in (track-name). <del> 6.14. </del> <ins> 6.13. </ins> The subscriber sends an "ANNOUNCE_CANCEL" control message to indicate it will stop sending new subscriptions for tracks within the provided

Related discussion:
text: I'm leaning towards CLOSED instead of CLOSE because CLOSE implies its a requested action, so to me SUBSCRIBE_CLOSE sounds like something the subscriber would send, not the publisher. For a similar reason, I like CANCEL slightly better than STOP, but both work. UNANNOUNCE and UNSUBSCRIBE are good words, but I think it's better to follow a pattern in this case and I like NAME sequence of messages.
NAME CANCEL and CLOSED are slightly better, sure. NAME there MUST still be an OK before objects can be sent. That way the receiver can tell upon receipt of the CLOSED if the subscription was ever active.
So to add an OK, we need to add another state. The question for me is what is the use case where we need to deliver that information in the end of the subscribe.
This discusion is probably easier looking at state machine diagrams
Sections 6.8 and 6.9 strongly imply that SUBSCRIBEFIN/RST are sent at the true end of the track. So are they not sent at the conclusion of a SUBSCRIBE that does not cover the end of the track? For example, if the last object of a track is (10, 12), and there's one subscribe for (4,0) to (5, 100) Is there a SUBSCRIBEFIN? If not, how does the subscriber know that in fact (5, 60) is the last object in that subscription? If so, does Final Group and Final Object refer to the last object in the subscription?
Individual Comment: Putting my HTTP hat on for a second, when you ask for something with unknown length, you infer from a graceful termination that the last thing you got was the true end of that resource. If you send a range request, the graceful termination at the end does not convey that the last thing was the true end. Once the object expires, you can issue another request to the same name and potentially get a totally different thing. Does this model work for moq, or do we need something different?
There are three different things you might want to communicate to a receiver at the end of the subscribe: 1) You got all of the objects in the range; 2) You got almost all of the objects, but some were dropped for latency/bandwidth reasons 3) The track was truncated for some reason and possibly, 4?
Code change:
Track Namespace: Identifies a track's namespace as defined in (track-name). <del> 6.14. </del> <ins> 6.13. </ins> The subscriber sends an "ANNOUNCE_CANCEL" control message to indicate it will stop sending new subscriptions for tracks within the provided

Related discussion:
text: 4) You got everything I have but i'm not sure if that's everything
is somewhat relevant to this issue.
As mentioned in , there are 3 different ways for a publisher to close a subscriber, each with minor differences. : I never sent you an object. : There are no more objects to send after group/object X/Y. : There are more objects after group/object X/Y... but I won't send them. A few things: I think it should be renamed since it's not the only ERROR message. Maybe ? The only difference is that one has an error code. Otherwise they both include a final group/object and it's up to the application/library to decide if they wait for the final object(s). I'd like to only implement this once but right now I have to copy/paste any code to handle these almost identical messages. What if we merged them and use error code 0 to signal a FIN. The reason phrase should be removed as part of . There really needs to be a state machine like with this many ways to terminate a subscription. I think these are all protocol violations: then then then My interpretation of the draft is that a followed by a is invalid, which is not obvious and unlike QUIC. I don't think so. The library/relay behavior doesn't change based on . The only thing that miiiight differentiate between the two is the application, which can use errors codes for this purpose. At the very least I would merge these two messages. But I would also like to explore a single way for the publisher to close a subscription. The application can signal the nuanced differences between errors (ex. error code 404 vs 410). The library/relay doesn't care, it just forwards the error code.
Individual Comment: The inspiration for FIN vs RST in moq came from HTTP caching. If a resource has no content-length, and the caching proxy has received N bytes, does it have the entire resource, or does it need to refetch the tail? If that response was terminated with a FIN (H2 or H3), then it has the whole resource. If it was terminated by a RST, it can't assume that it does. So the value I see in these two messages is for a moq relay is similar.
Code change:
Track Namespace: Identifies a track's namespace as defined in (track-name). <del> 6.14. </del> <ins> 6.13. </ins> The subscriber sends an "ANNOUNCE_CANCEL" control message to indicate it will stop sending new subscriptions for tracks within the provided

Related discussion:
text: A client connected to the relay and subscribed to a track, which fowarded the subscribe to a publisher. The track was published to the relay and ended with a FIN. Another client subscribes to this track from the beginning (Absolute/0,0) and no end. Does the relay need to resubscribe upstream? Now suppose that the connection from the publisher to the relay closed abruptly, or the track was terminated with SUBSCRIBE_RST. This is a signal to the relay that it's worth another subscribe to find out where the end of the track is. I can live if we consolidate the framing to a single message, and define a special error code to mean FIN, but I do think the transport needs to see the difference.
I like it. I would also like to merge ERROR with CLOSED, but we'll chat about that.