Code change:
Thanks, nice solution.
The latest draft of the Digest Fields RFC [0] deprecates all non-cryptographically secure algorithms (notably, MD5 and CRC32c). Given that the RFC is "not intended to be a general protection against malicious tampering with HTTP messages," I'd like to propose not deprecating at least these two algorithms, or changing "deprecated digest algorithms MUST NOT be used" to "should be used with caution" or similar. CRC32c is a very useful algorithm because it is composable (crc(A+B) == combine(crc(A), crc(B))). See [1] for an explainer and [2] for an implementation of combine() in zlib. MD5 is ubiquitous. AWS S3 and Google Cloud Storage both use it (see [3], which has similar syntax to this RFC except that the crc32c is base64 instead of hex). [0] URL [1] URL [2] URL [3] URL Sorry if this is the wrong forum. I saw some links from the w3c mailing list archive that looked like more discussions were happening here than there.
Hi NAME this is the right place, thanks for opening the issue. Do you have a use case where you'd like to use these algorithms with the Digest or Content-Digest header? They are pretty old and pretty bad at their intended purpose - what caution can we offer? The statement comes from URL, which is really about making it clear that applications do care about protecting messages also need to apply signatures that protect HTTP header or trailer sections. Otherwise, an attacker could just change the header at the same time as changing the representation.
Thanks for the fast reply. My use case is for detecting file corruption in internal HTTP communication, where we trust the network/server/client. Since it's internal, we don't have to obey a standard, but I was hoping to provide the CRC32C and MD5 that we already store to our end users -- provide something instead of nothing. Those algorithms provide an adequate level of error detection by our rough calculation (considering that TCP and other layers also checksum). Intel's implementation of CRC32C [0] runs at >19 GB/s, whereas SHA256 runs at only ~360 MB/s. We also take advantage of the ability of CRC32C to be combined, so we can concatenate files in O(1) time and get their CRC32C without calculating a SHA256/512 or MD5 over the whole content. We serve some millions of files. Google Cloud Storage and AWS S3 probably store many trillions of files and provide CRC32C+MD5 and MD5, respectively.1 It would be pretty impactful for servers to be able to pass along these digests and might make it worthwhile for browsers to start verifying digests for downloads (e.g. revive [1]). I might be under-appreciating the security intents of this RFC, but there is clear mention of the other benefits of the RFC (below), which are ones that non-cryptographically secure hashes are often good enough for. [0] URL [1] URL Edit: 1 Clarification: The S3 object is an MD5 in many scenarios, and S3 uses for integrity checks during uploads. They do not explicitly store the MD5 for all objects though.
Corruption in the network does happen, but HTTPS is a far better way of managing that risk. I know that you say that you trust them, but I would be worried about the network/server/client being - or becoming - untrustworthy. Obviously, if you are into high-volume transfers of bulk data, you are going to do what you need to do; specialized applications have specialized constraints. But that doesn't make unsecured transfers any less inadvisable; I'd stand by the deprecation here as a general statement.
Hi NAME Thanks for joining the discussion! Let me clarify one point: This just means that the part protected by Digest (the representation) is only a subset of an HTTP message. For the rest, while I agree on some of your considerations, eg. we had a very long discussion on that URL Maybe, since crc32 & co - while deprecated - are still defined in the digest algorithm table, I think that you wouldn't have particular interoperability issues if you need to use them in a specific environment.
Thanks NAME for the pointers to the earlier discussions. I'd thought I'd searched but somehow missed those. I'll yield, although I remain in favor of saying these "SHOULD NOT" instead of "MUST NOT" be used at least because of their ubiquity, performance and the composability of CRCs. One last question: does the Digest header provide better security than, or any additional beyond, what TLS provides? My reading of the RFC is that TLS is the typical primary protection layer (since doesn't protect headers; indeed, itself could be altered) and is primarily useful for finding ~bugs. (PS: I really like the digests, nice idea.)
NAME you've made some good points. Adding accommodation for these algorithms, given the proliferation of e.g. MD5 for this kind of integrity checking could be better than nothing, or hoping that everyone would run a SHA-algo over all of their existing assets. The thorny part is how to square the circle on relaxing the rule and providing clear, meaningful guidance to implementers. The MUSTNOT helps set a clear bar, this means its unambiguous what the spec thinks about the relationship of Digest to secure usage scenarios. Anything else needs careful wording and is hard to get right. The current MUSTNOT is slightly odd in so far as we don't state the ramifications if it is used. Part of that is a product of how Digest has been defined and how apps use it
NAME The actual work on digest was driven by its usage in signing http messages (eg URL and URL ) This is at a different layer wrt TLS. So you can see Digest as a building block for creating non-repudiable HTTP exchanges without altering the payload (since signatures are conveyed via http headers). Hth, R.
Personally -- I think there's something here worth considering. AIUI the primary use case for headers is not adversarial -- it's end-to-end (i.e., through various intermediaries, toolkits, perhaps a data store) integrity. If someone has lots of content they've calculated CRC or MD5 for, it seems reasonable to allow them to use it in ; not only is it more efficient, but it saves them recalculating a value for all of that content (and possibly getting it wrong -- that's effectively "end-to-end over time"). OTOH, there are adversarial use cases for too -- mostly, signing the digest as a proxy for signing the content. Here, using CRC or MD5 can be bad, and shouldn't be encouraged (or perhaps allowed) by the spec. This leads me to think that deprecating them isn't the right thing to do; instead, what about marking them as insecure in the registry, and putting language into the spec to explain how to use that? If folks think that's a good idea, I can do a PR.
Thanks for the background NAME . NAME NAME I think that sounds reasonable to state something like "insecure digests {MUST NOT be used|are not suitable} for protection against malicious actors." The final part of this I think would be allowing for the insecure digests.
I like NAME idea of introducing "insecure". It's not trivial how to state that insecure algorithms must not be used in signatures/adversarial use cases. I'm waiting for Mark's proposal :)
Maybe "Checksum Only" to imply a more narrow purpose. "Insecure" is not the opposite of "secure".
LGTMA pragmatic approach.

Related discussion:
text: Reference: UNIX, this document. <del> Status: deprecated </del> <ins> Status: insecure </ins>