jiebi/CodeConvo huggingface dataset:
We also take advantage of the ability of CRC32C to be combined, so we can concatenate files in O(1) time and get their CRC32C without calculating a SHA256/512 or MD5 over the whole content. We serve some millions of files. Google Cloud Storage and AWS S3 probably store many trillions of files and provide CRC32C+MD5 and MD5, respectively.1 It would be pretty impactful for servers to be able to pass along these digests and might make it worthwhile for browsers to start verifying digests for downloads (e.g. revive [1]). I might be under-appreciating the security intents of this RFC, but there is clear mention of the other benefits of the RFC (below), which are ones that non-cryptographically secure hashes are often good enough for. [0] URL [1] URL Edit: 1 Clarification: The S3 object is an MD5 in many scenarios, and S3 uses for integrity checks during uploads. They do not explicitly store the MD5 for all objects though.
Corruption in the network does happen, but HTTPS is a far better way of managing that risk. I know that you say that you trust them, but I would be worried about the network/server/client being - or becoming - untrustworthy. Obviously, if you are into high-volume transfers of bulk data, you are going to do what you need to do; specialized applications have specialized constraints. But that doesn't make unsecured transfers any less inadvisable; I'd stand by the deprecation here as a general statement.
Hi NAME Thanks for joining the discussion! Let me clarify one point: This just means that the part protected by Digest (the representation) is only a subset of an HTTP message. For the rest, while I agree on some of your considerations, eg. we had a very long discussion on that URL Maybe, since crc32 & co - while deprecated - are still defined in the digest algorithm table, I think that you wouldn't have particular interoperability issues if you need to use them in a specific environment.
Thanks NAME for the pointers to the earlier discussions. I'd thought I'd searched but somehow missed those.
I'll yield, although I remain in favor of saying these "SHOULD NOT" instead of "MUST NOT" be used at least because of their ubiquity, performance and the composability of CRCs. One last question: does the Digest header provide better security than, or any additional beyond, what TLS provides? My reading of the RFC is that TLS is the typical primary protection layer (since doesn't protect headers; indeed, itself could be altered) and is primarily useful for finding ~bugs. (PS: I really like the digests, nice idea.)
NAME you've made some good points. Adding accommodation for these algorithms, given the proliferation of e.g. MD5 for this kind of integrity checking could be better than nothing, or hoping that everyone would run a SHA-algo over all of their existing assets. The thorny part is how to square the circle on relaxing the rule and providing clear, meaningful guidance to implementers. The MUSTNOT helps set a clear bar, this means its unambiguous what the spec thinks about the relationship of Digest to secure usage scenarios. Anything else needs careful wording and is hard to get right. The current MUSTNOT is slightly odd in so far as we don't state the ramifications if it is used. Part of that is a product of how Digest has been defined and how apps use it
NAME The actual work on digest was driven by its usage in signing http messages (eg URL and URL ) This is at a different layer wrt TLS. So you can see Digest as a building block for creating non-repudiable HTTP exchanges without altering the payload (since signatures are conveyed via http headers). Hth, R.
Personally -- I think there's something here worth considering. AIUI the primary use case for headers is not adversarial -- it's end-to-end (i.e., through various intermediaries, toolkits, perhaps a data store) integrity. If someone has lots of content they've calculated CRC or MD5 for, it seems reasonable to allow them to use it in ; not only is it more efficient, but it saves them recalculating a value for all of that content (and possibly getting it wrong -- that's effectively "end-to-end over time"). OTOH, there are adversarial use cases for too -- mostly, signing the digest as a proxy for signing the content.
Here, using CRC or MD5 can be bad, and shouldn't be encouraged (or perhaps allowed) by the spec. This leads me to think that deprecating them isn't the right thing to do; instead, what about marking them as insecure in the registry, and putting language into the spec to explain how to use that? If folks think that's a good idea, I can do a PR.
Thanks for the background NAME . NAME NAME I think that sounds reasonable to state something like "insecure digests {MUST NOT be used|are not suitable} for protection against malicious actors." The final part of this I think would be allowing for the insecure digests.
I like NAME idea of introducing "insecure". It's not trivial how to state that insecure algorithms must not be used in signatures/adversarial use cases. I'm waiting for Mark's proposal :)
Maybe "Checksum Only" to imply a more narrow purpose. "Insecure" is not the opposite of "secure".
LGTMA pragmatic approach.
This removes the dictionary-specific ttl and switches to using the dictionary resource freshness as the mechanism for expiring dictionaries.
NAME could you please take a look and see if this looks ok? NAME could you look at the language around the freshness of the resource relative to cache-control and stale-while-revalidate and make sure I expressed it properly (and it is something we're ok with leveraging)?
Why does the dictionary lifetime need to be independent of the cache lifetime? Having an object that is cached with a separate lifetime seems unnecessary. I would have said that clients can cache a dictionary for as long as they want (as always), but no longer than the lifetime stated in the cache-control field. Now, you might say that a server might want a resource to be usable in its own right (i.e., as an ordinary resource) for one duration, with a separate lifetime for its use as a dictionary. I see two reasons that this isn't much of a compelling argument: A server can forget about a dictionary and not provide delta compression at any time, so there is no real need for the dictionary lifetime to be shorter than the cache lifetime. Client caches won't want to retain something for longer than its validity, so I don't see much chance that a resource will be kept past its cache lifetime. Resources like this are likely exclusively used as a dictionary. There isn't much value for those resources having a separate lifetime in that case.
The main reason for a separate lifetime for the dictionaries is to allow for sites to reduce the possible variations that are in the wild and reduce the number of variations of that come in and constrain the number of previous releases that they need to compress artifacts for. Otherwise, a dictionary "miss" will result in a full version of the resource getting added to caches for every possible version of where delta-compressed artifacts aren't available. Caches can probably manage that OK on their own, but by reducing the dictionaries to a known window it both provides a lever of control on the variations and also allows for clients to "hit" on a cached version of the resource when they have an old dictionary.
This has been rattling around in the back of my head for the last month or so and I wanted to get my thoughts down for discussion.
I think it might be possible to use the existing validity of max-age and stale-while-revalidate to achieve the same result but I'm a bit worried about the complexity that would bring to users. There are 2 main scenarios that express the tension: Versioned immutable resources that change the URL when the content changes. Well-known URLs that are cacheable but update the content in-place (libraries for analytics, ads, embeds, etc). For the purposes of this example, lets assume that the resources are updated twice a day and that 80% of repeat traffic re-visits within 10 days of their previous visit. Assuming a serving architecture with a server-side cache (CDN, load balancer, etc): In both cases, the response is cacheable and keyed based on the dictionary that the client sends as available (). The main concern is how to tune the requests so that the cache behaves like it does today, minimizing traffic to the origin. The "current" version of the resource will be the only one actively fetched by the application independent of when the user last visited. The current best-practice is to cache them for a long time (1 year) and mark them as immutable. Without dictionaries that means that the cache will effectively only see requests for the current version of the resource and all previous versions can age out of the cache (even if they have a long time-to-live, just by virtue of never being accessed). There will be one request to the origin (or a small number) while the cache gets populated but all future requests will always be served from the edge cache. With dictionaries, the cache key depends on when the client last visited the page since the dictionary will be the version of the resource they have in their cache. That means 80% of the requests will be covered by 21 cache keys (one without a dictionary and 20 delta-compressed variants) but that there will be 700+ variants in the wild if the 1-year resource expiration is reused for the dictionary. The cache could manage access frequency to store just the 21 "hot" variants for the last 10 days but the 20% of visits that are using older dictionaries will result in cache misses that require going back to the origin. If a developer shortens the resource cache lifetime to a shorter time, like 5 days, knowing their release schedule then the problem largely goes away.
That would require tying the dictionary use to the expiration of the resource and not allow for it to be used even if it is still in cache (but expired). That would mean we are now imposing a short cache lifetime on an immutable resource in order to optimize the hit rate of the middle caches. It feels like explicitly managing the dictionary lifetime at the same place where the dictionary details are specified might be cleaner. For well-known URLs that are not immutable and change the content in-place, we want to be able to use the previous version(s) of a resource as a dictionary for the "current" version. To be able to do that we need the dictionary to be viable beyond the expiration time of the resource. Since we need to hard-clip the use of the dictionaries to prevent cache key issues we'd need to have the dictionaries use something like the time window to allow for a revalidation request of an expired resource while still using the dictionary but allowing for a hard-clip of the dictionary validity at the end of the stale-while-revalidate time. It's theoretically possible but it feels like we're tying the dictionary TTL into the cache heuristics in ways that might not be intended and might also be easy to get wrong.
"well-known" is a term of art you might want to avoid here. I get your intent. It looks like there is a fairly good suggestion for . It might take a bit of time for a cache to be able to work out the length of time from the last access that represents a good time horizon for use. But that seems manageable. The resources that churn is an interesting one. It seems like you are assuming that the cache lifetime is short, such that a replacement version is requested AFTER the cache lifetime expires. Your option does seem like a possible way of managing this, but I'd be unsure whether stale resources are good for use. You'd have to specify that behavior (reasonable, but it would need to be specified). It might be that a longer cache lifetime also works well enough. Clients do tend to re-check earlier than expiration in many cases, which would ensure that the resource is fresh in use, while making the overlap work well.
NAME NAME does using the resource max-age + stale-while-revalidate as and expiration for the client-side dictionary work ok for you (with a recommendation that immutable resources specify expirations based on their typical release cycle)?
I think that having an explicit life time for the dictionary is a little bit more flexible than relying on cache-control headers, but I think that by itself should outlive most dictionary uses I can think of, especially given that the CDN is unlikely to keep those resources in its own cache beyond that period as well.
NAME I'm not sure I understand (or we're not talking about the same part of the issue). The main concern for caches is how long clients consider a dictionary valid for which will dictate how long of a window a given resource will see request headers from. e.g. for a dictionary with a max-age of 1 year, clients will advertise that dictionary as being available for up to a year. If there are 1,000 updates to the resource used as a dictionary (not impossible for a js app that has 3 releases per day) then there will be 1,000 variants of the dictionary in the wild at any given time (with more frequent requests for more recent dictionaries but still some infrequent long-tail requests). To prevent cache misses, it would be beneficial to cap the dictionary validity to a smaller window where most return traffic falls in and have older clients download the full resource from cache rather than missing and going back to the origin. It also caps how many delta-compressed versions need to be created. The discussion point is if that "client valid" time for a dictionary should be its own separate value or if sites should adjust their max-age to be more reasonable instead if they want to limit the variants (capping the validity to the max-age but requiring a shorter max-age rather than blindly setting everything to 1-year).
Worst-case, a separate ttl could be added later if using the resource expiration ends up being a problem blocking adoption.
It seems like there are two different mental models of how caching and cache-misses could work. The difference between them may be contributing to the confusion. The first is that the CDN is keeping "resource A compressed against dictionary X" as a cache item, and if a client makes a request for "resource A compressed against dictionary Z", the CDN needs to go back to origin to retrieve that combination. The second is that the CDN is keeping "resource A" and "resource Z" as cache items, and if a client makes a request for "resource A compressed against dictionary Z", the CDN either has Z locally or not.
If not, it will compress resource A in a different way the client supports. (Of course, this doesn't prevent the CDN from having some amount of second-layer caching to avoid recompressing frequent combinations.) Personally, I tend to think the second tends to have better properties -- you don't have an explosion of cache entries or origin requests for the long tail, you simply fall back to a non-dictionary compression scheme. CDNs might choose to count usage as a dictionary as a cache hit for Z, making it stay in cache as long as it continues to be used by clients, but might not choose to fetch Z again if it ages out of the cache. (The interesting quirk from a CDN perspective is that a single resource might have multiple old versions still getting "hits.") The two then probably imply different ways of thinking about lifetimes. In the first case, you could want a separate lifetime on the dictionary to feed into how long the CDN needs to keep around the f(A,Z) item. In the second, the lifetimes of the individual objects likely suffice. (For that matter, if the client is identifying the version of Z it has, then it doesn't really matter whether that object is fresh or not, only whether the server still possesses the indicated version.)
The main use case for this is for something like a web app at build time where they pre-compress the resources at build time for some number of previous releases. At serving time, the CDN doesn't have any special dictionary logic and just honors the headers to vary on and the origin checks for pre-compressed assets using the requested dictionary (in my apps that I was doing this with, I just added the hash to the end of the file name and did a quick file check). On a miss, the origin sends the full resource which ends up in a separate cache key on the CDN. This allows for origins to support dictionary-compressing the content without their CDN or load balancer having to have implemented it.
Right, that's a reasonable workaround for CDNs that don't know about this spec. But wouldn't we rather the CDNs could implement the spec and calculate the delta at the edge? Wouldn't that be the desired end state? Perhaps I'm wrong, or perhaps I've wandered into an architectural question bigger than this issue.
I'm not 100% sure it's just a workaround and origin-compressing against previous releases at build time may be preferable for some cases. The most extreme case that comes to mind is something like a huge WASM app (like Photoshop) where the files are upwards of 50MB (well, huge relative to the usual web page resources). Presumably the CDN could still do the work to do the compression (maybe as an offline job for future requests) so it doesn't require the origin to do it necessarily but it's a lot more reasonable in the extreme cases. That said, I think we can still get buy without a separate TTL and adding one later if it ends up being useful can be done as an enhancement without breaking the existing case.
I like red patches. Thanks.
This is awkward, but this is better.
and 8.8.4: Because "control data and a request header and message content" reads a bit oddly, we changed it to "control data and a request header with message content" per "control data and a request header with no message content" in Please let us know if this is incorrect. Original: An HTTP POST request that includes control data and a request header and message content is transmitted as one HEADERS frame, followed by zero or more CONTINUATION frames containing the request header, followed by one or more DATA frames, with the last CONTINUATION (or HEADERS) frame having the ENDHEADERS flag set and the final DATA frame having the ENDSTREAM flag set: ... A response that includes control data and a response header and message content is transmitted as a HEADERS frame, followed by zero or more CONTINUATION frames, followed by one or more DATA frames, with the last DATA frame in the sequence having the ENDSTREAM flag set: Currently: An HTTP POST request that includes control data and a request header with message content is transmitted as one HEADERS frame, followed by zero or more CONTINUATION frames containing the request header, followed by one or more DATA frames, with the last CONTINUATION (or HEADERS) frame having the ENDHEADERS flag set and the final DATA frame having the ENDSTREAM flag set: ... A response that includes control data and a response header with message content is transmitted as a HEADERS frame, followed by zero or more CONTINUATION frames, followed by one or more DATA frames, with the last DATA frame in the sequence having the ENDSTREAM flag set:
The close connection item used to be defined in Connection as whereas now its effect is described in Messaging without definition. I think it should be defined somewhere.
gets pretty close to that text: ... with text below that elaborates. What do you want to see here?
looks good except as noted by NAME the bare close strings look weird.LGTM, except for 'the close' being a bit weird (but I don't feel that strongly about it; suggestions for replacement inline).
The current text of "ought to ... where possible" is easy to read as saying "if you have the technical capability, do it". There are, however, some subtleties, such as if you get the 308 response over http-not-s, in which case the question of whether or not to re-link is not so clear-cut. Change from "where possible" to "where appropriate" to hint that there is some logic needed here beyond "blindly accept". (inspired by follow-up discussions from )
I like Roy's version better than mine :) I guess I was too conservative...
I think that begs a follow-on question about when it is appropriate. I would be more comfortable with changing the entire sentence to be The change itself is good, but it needs to be done for 301 as well.
The specification of can be read to allow the response to be reused for another concurrent request, since it's specified in terms of non-volatile memory. Likewise, other cache directives are silent on such reuse. So-called "request collapsing" has become quite common in proxies, reverse proxies and CDNs. Clarification as to whether such reuse for temporally concurrent requests would be helpful (in the case of and otherwise).
This also seems related to and , which allow the response to be re-used after cache control would say not to re-use it: URL
Discussed in Montreal; introduce the concept of a single-use response, and carve out no-store.
Includes several document cleanups for signatures, to publish as -09
Also: If there is no query string should the corresponding signing text be or ie. should the query value be empty or just consist of one Should we return with this request and return on this request This is what I opted for in my implementation. See shows that http4s makes the distinction between blank queries and empty queries is simpler I think this is a corner case that is worth explicating very clearly, perhaps even with a test.
There are quite a few occurrences of the header in the draft. From an implementation point of view, Go "swallows" the header and it is not part of the standard Request structure. There's a good reason for the Go library to do it, and in fact the draft recommends to use NAME instead. Can we remove from examples, e.g. Sec. 2.1, so we don't train people to do the wrong thing?
There are only a couple instances of the "host" header in the current draft left, and one of them (section 2.1) is specifically about how to format headers. It could be removed from the other example signature input strings without affecting anything negatively. Same with the "date" header, which could be confused with the "created" parameter, see
This sentence is correct: But it's true for most reasonable uses of , and people will still be using this parameter. So why not add:
No, we explicitly did not want to conflate this with the jwa registry. We also didn't want to have a second tier choice like having "alg=jose" and then having the actual algorithm elsewhere, since a big problem with late editions of the Cavage draft were exactly that process. (See the mess around "hs2019"). The editors discussed this at length and decided the best compromise was to define a way that jwa algorithms could be used in an interoperable fashion, but not to allow runtime signaling with them. Better to have the choice of jwa be signaled within the application level. Additionally, it's preferable to not use the runtime parameter to signal the algorithm any way, which is discussed at length in the security considerations.
All good, so why aren't you using normative language?
It's not normative because this section defines the algorithms, not the parameter.