jiebi/CodeConvo huggingface dataset:
And if the new resource is really much larger than the old one, the benefits of delta compression are dubious anyway. So the suggestion is to authorize a window size which is slightly larger than the external dictionary size. As to how much larger exactly, a suggestion here is by +25% (x1.25). I would expect a large majority of update scenarios to fit into this picture, with the new resource being slightly larger than the old one by a few %. For the less common cases where the new ressource is much larger than the old one, Zstandard could still employ the dictionary but only for the beginning of decompression. The compression process will have to select a window size which remains under the specified limit (x1.25 external dictionary size). As a technical insight, note that byte-accurate window size is only a thing when the window is the entire decompressed content. Otherwise, the window size is encoded in a condensed format, which can only represent increments by 1/8th of a power of 2. This is not a problem to determine this "largest window size under the specified limit", but due to limited accuracy, you can already guess what would be the problem is we were to keep the formulation "not more than the larger of 8 MB and the size of the dictionary". Imagine that the external dictionary size is bytes, and the new resource to delta-compress is , aka 1-byte larger than the dictionary. With the initial formulation, since the window size is limited to the size of the dictionary, the compressor cannot use as a window size, because it's 1 byte larger than the limit. So, it will have to find the next best window size under the limit. Due to granularity limitations, this next best window size is 8 MB. So delta compression would proceed fine for the first 8 MB, but for the last 1 MB, data would have to be compressed without the benefit of the dictionary. With the newly proposed limitation to "x1.25 the dictionary size", delta compression would remain active in above scenario during the entire decompression process. And it would remain fully active if the new resource to decompress is 10 MB, 11 MB or 11.24 MB.
But if the new resource is 11.25 MB or larger, then the compressor will automatically limit the window size to 11 MB (since it's the largest representable window size under the limit, which, in this particular instance, is ), achieving the wanted memory size control.
Since neither the server not the client may know what the resource size will be when compression starts (say, in a streaming HTML case), won't the 1/8th power of 2 granularity apply even for cases where the resource is the same size (or sometimes smaller) than the dictionary? For resources slightly larger than the old one, the dictionary would still be used for the majority of the resource but the last bit would fall back to regular window-based compression. I agree that having the dictionary available for the whole resource would provide better compression but I'm not sure where to draw the line. Something like a 25% fudge factor can be quite a bit when you start talking about 50 MB+ resources that might be close to where the client wants to draw the line anyway. They may have allowed the dictionary at 50 MB but not allow it's use at all if it needed 62 MB available for a window. A similar case could be made for the non-update case where you have a custom-built dictionary but the resource being compressed exceeds 8 MB. Only the first 8 MB of the resource would be able to use the dictionary. Hopefully at that point the compression window would have most of the stuff that was in the dictionary but the dictionary becoming unavailable is something we should expect for some cases with ZStandard, even if we use a fudge factor. Is there a way to convey the 1/8th power of two granularity on the window size or is it something that the ZStandard libraries will take care of automatically for users (i.e. setting the window size to 11 MB when the caller sets it to 11.24 MB)? I'm wondering how much of this implementors are going to have to deal with directly.
It depends if the compression side (server) is informed upfront of the size of the payload to compress. If not, then indeed you are right, the compressor will have to set a Window Size, and the granularity rule applies. So for example, let's assume the payload to delta-compress is 50 MB, and the reference dictionary is also 50 MB.
Let's then assume it implies a window size limit of 50 MB (dictionary size) and the compressor is not informed of the size of the payload. In which case, the compressor will have to set a Window Size, and it will have to settle for the largest representable window size under the limit, which happens to be 48 MB. Therefore, the dictionary won't be used during the compression of the last 2 MB of the payload. With the proposed +25% tolerance rule, the limit would become 62.5 MB, and the largest representable window size under the limit is 60 MB. Now the dictionary remains active during the entire compression of the 50 MB payload. When the data to decompress ends up being smaller than the window size, it's obviously not a problem for the decompression process: the final decompressed size is still properly reported at the end. It's just that the intermediate buffer allocation will be larger than it could have been. Note that uses for the allocation of buffers, so the address space is just reserved, but the physical memory may never be used if it's not needed (i.e. if there is no write action on the later segment part). +25% is more like a suggestion, this can certainly be discussed if some other limit feels better. I liked that it's a (binary) clean factor, and it's larger than , which covers the scenario above, and gives a little bit of breathing room if the newer payload is a bit larger than the older reference one. I see this scenario as being very distinct from delta-compression. For delta-compression, we expect large segments to be identical between the reference and the payload to compress, so we want this ability to access the reference during the entire compression of the payload. The price to pay is that the reference is very specific, and is not designed to offer any serious benefit to any other payload. Custom dictionaries are different: they are supposed to carry the most relevant "commonalities" detected across a set of samples. Presuming a large enough sample set, the common components are not expected to be very large. In such a scenario, the custom dictionary is expected to be (relatively) lightweight (compared to delta-compression), likely in the <= 100 KB range.
Consequently, by the time the payload to compress reaches 1 MB, it should already contain all references it needs to compress the rest of the file. I would not expect a large impact of the custom dictionary beyond that point, let alone beyond 8 MB. The plan would be for the library to do this adaptation automatically. The user would not have to know anything about it, though it's better if the user understands that the library selects the closest window size which is <= to the limit, so that they are not surprised if, upon inspection, the final window size is not byte-identical to the set limit. A good API documentation will likely help to reach this objective If by implementors you mean "server side programmers", they would have to know what's the implicit window size limit when a client requests a format (essentially the rule you are currently defining). Then they would set this limit as a compression parameter, and the library will take care of the adaptation into window size.
Yes. (Keeping in mind of course that most traffic in question will presumably be under the 8 MB floor and so none of this complexity will apply.) Right now the zstd library has limited controls about setting detailed window constraints. We'll need to add a new option to give users that fine-grained control. We can do that with an eye towards user simplicity. Frankly, I'm also very open to also adding a flag (and equivalent compression parameter in the library) that would just look at the size of the dictionary (which zstd necessarily has access to) and would configure the window accordingly. In that case, users wouldn't have to do any math at all.
OK, I chaged the structure and language so that they are no longer streams of bytes compressed with a given algorithm but are now specifically and . The references for those named formats have the window restrictions (which for Zstandard includes the 1.25x dictionary requirement). Currently the byte stream itself is still a native Brotli/Zstandard stream, just with specific settings but this also makes it cleaner if we decide to . PR with the update is here: URL
lgtm with nits, thank you!LGTM!
An alternative to - removes HTAB everywhere, replaces with .
Right now we use OWS, and suggests moving from SP to RWS for inner list separation. Should we allow tabs in SH? Error on them? If we want to allow them, 983 makes sense, as it's more consistent. However, we'll need to test for tabs...
What's the reason to allow tabs? Compatibility with current headers? Do actual tab characters show up in the wild that often?
We had them in there because we were reusing OWS, not purposefully. AFAICT they don't appear in the wild much at all. So +1 for .
LGTMThank you for working on this. LGTM.
Into SUBSCRIBEDONE. I didn't remove the SUBSCRIBEERROR message in favor of SUBSCRIBE_CLOSED as NAME suggested in this PR. Addresses some comments in , but possibly not all
So my notes from this discussion. CLOSE is confusing with QUIC layer. People seemed to prefer UNSUBSCRIBE over SUBSCRIBE_CLOSE. I love the move aware from RST and FIN as naming.
Propose names from my understanding of discussion call Announce (send by publisher) , AnnouceOK, AnnouceError UnAnnounce (send by publisher) UnAnnounceOK, UnAnnounceError AnnouceEnd ( sent by thing that received Annouce) , AnnouceEndOK, AnnouceEndOK Subscribe (send by subscriber) , SubscribeOK, SubscribeError UnSubscribe (send by subscriber) , SubscribeOK, SubscribeError SubscribeEnd (send by thing that received subscriber), SubscribeOK, SubscribeError Mostly trying to get rid of FIN / RST and have different names for the messages that flow in opposite directions.
I don't like the name of the different messages used to terminate. I'd rather have one message that transitions to the terminal state, with an optional error code and final group/object. IMO the subscription flow: And the ANNOUNCE flow mirrors: The brackets mean a message is optional. The sender is responsible for creating and the receiver is responsible for closing, possibly at the bequest of the sender (like QUIC's STOP_SENDING).
has been merged, adding yet another term (CANCEL), but I figured we'd clean that up with this issue.
Since we are bikeshedding, I prefer ANNOUNCE_RESCIND instead of CANCEL, since an announce is more of a declarative statement than an active request from the sender.
Heh and UNANNOUNCE and UNSUBSCRIBE stick out because they don't share the same prefix.
I like the names being proposed as they help make make is easier to understand. I do want to separate out one issue. The STOP / CLOSE type thing happens way after the initial SUBSCRIBE thus it can end up in a situation where auth has expired or something else has gone wrong this I like the current design that allows an OK or ERROR to be returned for those. I think it is also key for sender of the things stoping the subscribe to know that the other side has acted on that so we need the OK.
I'm leaning towards CLOSED instead of CLOSE because CLOSE implies its a requested action, so to me SUBSCRIBE_CLOSE sounds like something the subscriber would send, not the publisher. For a similar reason, I like CANCEL slightly better than STOP, but both work. UNANNOUNCE and UNSUBSCRIBE are good words, but I think it's better to follow a pattern in this case and I like NAME sequence of messages.
NAME CANCEL and CLOSED are slightly better, sure. NAME there MUST still be an OK before objects can be sent. That way the receiver can tell upon receipt of the CLOSED if the subscription was ever active.
So to add an OK, we need to add another state. The question for me is what is the use case where we need to deliver that information in the end of the subscribe.
This discusion is probably easier looking at state machine diagrams
Sections 6.8 and 6.9 strongly imply that SUBSCRIBEFIN/RST are sent at the true end of the track. So are they not sent at the conclusion of a SUBSCRIBE that does not cover the end of the track? For example, if the last object of a track is (10, 12), and there's one subscribe for (4,0) to (5, 100) Is there a SUBSCRIBEFIN? If not, how does the subscriber know that in fact (5, 60) is the last object in that subscription? If so, does Final Group and Final Object refer to the last object in the subscription?
Individual Comment: Putting my HTTP hat on for a second, when you ask for something with unknown length, you infer from a graceful termination that the last thing you got was the true end of that resource. If you send a range request, the graceful termination at the end does not convey that the last thing was the true end. Once the object expires, you can issue another request to the same name and potentially get a totally different thing. Does this model work for moq, or do we need something different?
There are three different things you might want to communicate to a receiver at the end of the subscribe: 1) You got all of the objects in the range; 2) You got almost all of the objects, but some were dropped for latency/bandwidth reasons 3) The track was truncated for some reason and possibly, 4?
4) You got everything I have but i'm not sure if that's everything
is somewhat relevant to this issue.
As mentioned in , there are 3 different ways for a publisher to close a subscriber, each with minor differences. : I never sent you an object. : There are no more objects to send after group/object X/Y. : There are more objects after group/object X/Y... but I won't send them. A few things: I think it should be renamed since it's not the only ERROR message. Maybe ? The only difference is that one has an error code. Otherwise they both include a final group/object and it's up to the application/library to decide if they wait for the final object(s). I'd like to only implement this once but right now I have to copy/paste any code to handle these almost identical messages. What if we merged them and use error code 0 to signal a FIN. The reason phrase should be removed as part of . There really needs to be a state machine like with this many ways to terminate a subscription. I think these are all protocol violations: then then then My interpretation of the draft is that a followed by a is invalid, which is not obvious and unlike QUIC. I don't think so. The library/relay behavior doesn't change based on . The only thing that miiiight differentiate between the two is the application, which can use errors codes for this purpose. At the very least I would merge these two messages. But I would also like to explore a single way for the publisher to close a subscription. The application can signal the nuanced differences between errors (ex. error code 404 vs 410). The library/relay doesn't care, it just forwards the error code.
Individual Comment: The inspiration for FIN vs RST in moq came from HTTP caching. If a resource has no content-length, and the caching proxy has received N bytes, does it have the entire resource, or does it need to refetch the tail? If that response was terminated with a FIN (H2 or H3), then it has the whole resource. If it was terminated by a RST, it can't assume that it does. So the value I see in these two messages is for a moq relay is similar.
A client connected to the relay and subscribed to a track, which fowarded the subscribe to a publisher. The track was published to the relay and ended with a FIN. Another client subscribes to this track from the beginning (Absolute/0,0) and no end. Does the relay need to resubscribe upstream? Now suppose that the connection from the publisher to the relay closed abruptly, or the track was terminated with SUBSCRIBE_RST. This is a signal to the relay that it's worth another subscribe to find out where the end of the track is. I can live if we consolidate the framing to a single message, and define a special error code to mean FIN, but I do think the transport needs to see the difference.
I like it. I would also like to merge ERROR with CLOSED, but we'll chat about that.
The issue discussion also mentioned the ability for the server to indicate a GET-able URI for the returned results, presumably using Content-Location. It looks like the definition in -semantics is probably sufficient, but it's probably worth highlighting that there are caching implications to having that header present in one of these responses.
Content-Location doesn't have any effect on caching. It would be odd to add caching implications to it.
NAME I was recalling this: URL thinking that the response should be cached for the indicated resource. But I had missed this: And indeed, the Caching spec talks only about invalidating any existing entry for the indicated resource, rather than giving any permission to cache the response under that URI. The application is allowed to update its local copy based on Content-Location, but apparently caches aren't.
We should figure out/document: 1) Establishing the cache key - a cache needs to know how to compute a cache key for the request. The obvious (and very inefficient) way to do this is to use the entire request body as part of the key. 2) Request canonicalisation - we should allow caches with knowledge of canonicalisation algorithms for a given format to use them. 3) Indicating canonical form in requests - if a client (UA or intermediary) canonicalises, it would be nice if they could assert that in the request, so a downstream cache can assume it's there and just look for a byte-for-byte match, rather than re-canonicalising. This would allow a client to manufacture cache misses, but that's already possible in URLs... 4) Indicating cache key in requests - E.g., a hash of the request body. Needs security analysis, though. 5) Indicating cache key in responses - If the server can instruct caches as to what future requests would match it, that would be very advantageous. Necessarily requires knowledge of the request format, but we could define common ones for url encode, XML, JSON.
I don't think that (4) is generally feasible. There might be cases where a cache trusts other entities sufficiently that a hash (Digest field?) can be used, but it will need to work this out for itself. This is probably the big cost associated with caching this method.
As for the general case of c14n, it's unfortunate that this is necessary, but it is probably better than relying on having some shared understanding of an abstract model for the resource representation. I would not build this on any assumption that XML c14n is possible. A complete solution there is likely impossible. I know less about the warts of JSON, but even that is entering into dangerous territory. You might get more traction with schema-aware c14n with the understanding that it is not generally applicable to any document. That is, define tools so that specific XML- or JSON-based formats can opt in to the use of c14n. People might then apply that c14n if the format allows it; people might also apply the c14n if the format does not, but we have not promised them anything and so guarantees might not apply.
nod there's going to be a tradeoff between the amount of knowledge/processing necessary for c14n and cache efficiency. It might be good to explore what conventions a format needs to follow to make it easily canonical; I can't help but think that most use cases for this are going to be defining a new query language, not reusing an existing document or even data format.
I think that caching is a "must have", though probably we could just provide minimal hints and leave clients the onus of hitting the cache (eg. clients c14n-izing requests in some may increase cache hits). I won't mess with the request media-type (json? xml? x-www-form-urlencoded? ...) and just use the content checksum (eg. ). I think that supporting media-type aware caches is really hard to implement (eg. see rfc8785 and rfc7493 only for json): there's room for experimental specs though... That's ok, though see (2) about supporting media-types is hard Agree wrt Security Considerations IF we want to support a caching based on a response cache key, this could be implemented with some caveats without re-sending the at all: Authentication information should be processed; MUST NOT contain authnz information
Caching will be tricky for "similar" requests sent by different clients. On the other hand, caching could be simple for requests that are repeated by the same client (like in a refresh operation).
(opening this so it has a discussion venue) Currently, the spec uses "SEARCH" (for the reasons given in the spec). Alternatives: 1) pick a new name 2) use GET with payload (probably unrealistic, just listed for completeness) 3) use POST and find a way to label it as safe 4) re-use another existing safe method that takes a body (PROPFIND, REPORT) (just listed for completeness) The obvious drawback in "pick a new name" is the forseealke bike-shedding.
Just my .02. Using is initially attractive because it's familiar; effectively we'd be extending the semantics of the method with a header. However, I'm not sure it's a good idea. implies certain guarantees to clients, servers and intermediaries -- e.g., write-through. If an extension can change those guarantees, it's important that they're visible to all components, and an extension (e.g., a header) by nature can't guarantee that. It seems to me that one of the critical things to do is to get a positive indication from the user agent that the request they're sending doesn't have write-through, so a extension would need to carry an indication from the client. However, the origin server is also likely to need to convey a cache key and perhaps additional information, which leads this approach to something that's looking complex; the client has to send a certain header, the server has to send a certain header, and an intermediary still might not understand what's going on (which might be good or bad). So that makes me lean towards the status quo, (1), or (4) as being more straightforward. Reusing an already-deployed name has been discussed elsewhere; I'm not convinced that it's going to add a lot of value, considering that most traffic is HTTPS now and so intermediaries with fixed lists of supported methods are much less common. If we had data that they'd get through e.g., an enterprise virus scanning proxy whereas a new method wouldn't, I'd be interested to see that. Still, if fits the bill adequately, I think it's fine. The real test will be whether folks (especially HTTP API developers) find it compelling enough to use; I'm not sure quite gets there, but it's in the ballpark. If we're going to consider new names, I'd suggest .
If the point of this exercise is to make safe-ness visible to intermediaries, then a method is the right place to put that signal. I'm comfortable enough with , though an entirely name would be equally good. (I think that we should continue discussion about cache keys separately.)
I value a lot the ability for intermediaries to filter out write requests before they reach an application. Reusing won't have this advantage.
Interim discussion: switch document to use , which has good support from the attendees, and see how it seems to people who would be adopting.
Que? This is far more than the simple rename I was expecting. I had a similar double-take on first read, but most of the additional changes appear to be excising the WebDAV references, which is reasonable to do along with the rename.