Code change:
that are not authoritative (as defined in RFC7540, 10.1) for the indicated origin. <del> CACHE_DIGEST allows the client to indicate whether the set of URLs used to compute the digest represent fresh or stale stored responses, using the STALE flag.  Clients MAY decide whether to only send CACHE_DIGEST frames representing their fresh stored responses, their stale stored responses, or both. </del> Clients can choose to only send a subset of the suitable stored <del> responses of each type (fresh or stale).  However, when the CACHE_DIGEST frames sent represent the complete set of stored responses of a given type, the last such frame SHOULD have a COMPLETE flag set, to indicate to the server that it has all relevant state of that type.  Note that for the purposes of COMPLETE, responses cached since the beginning of the connection or the last RESET flag on a CACHE_DIGEST frame need not be included. CACHE_DIGEST can be computed to include cached responses' ETags, as indicated by the VALIDATORS flag.  This information can be used by servers to decide what kinds of responses to push to clients; for example, a stale response that hasn't changed could be refreshed with a 304 (Not Modified) response; one that has changed can be replaced with a 200 (OK) response, whether the cached response was fresh or stale. </del> <ins> responses.  However, when the CACHE_DIGEST frames sent represent the complete set of stored responses of a given type, the last such frame SHOULD have a COMPLETE flag set, to indicate to the server that it has all relevant state of that type.  Note that for the purposes of COMPLETE, responses cached since the beginning of the connection or the last RESET flag on a CACHE_DIGEST frame need not be included. CACHE_DIGEST will also include the cached responses' ETags, if they were present in the response.  This information can be used by servers to decide if a response needs to be pushed to clients; If a response is cached and was not changed at the origin server, the server calculating its hash will find it in the digest and therefore will not push it.  If a response is cached but was modified at the origin server, the server calculating its hash will not find it in the digest, so the response will be pushed. </ins> CACHE_DIGEST has no defined meaning when sent from servers, and SHOULD be ignored by clients. 2.1.1. <ins> Given the following inputs: * "P", an integer smaller than 256, that indicates the probability of a false positive that is acceptable, expressed as "1/2\*\*P".  * "N", an integer that represents the number of entries - a prime number smaller than 2**32 Let "f" be the number of bits per fingerprint, calculated as "P + 3" Let "b" be the bucket size, defined as 4. Let "bytes" be "f"*"N"*"b"/8 rounded up to the nearest integer Add 5 to "bytes" Allocate memory of "bytes" and set it to zero.  Assign it to "digest-value". Set the first byte to "P" Set the second till fifth bytes to "N" in big endian form Return the "digest-value". 2.1.2. </ins> Given the following inputs: <del> "validators", a boolean indicating whether validators (RFC7232) are to be included in the digest; </del> <ins> "URL" a string corresponding to the Effective Request URI (RFC7230, Section 5.5) of a cached response </ins> <del> "URLs'", an array of (string "URL", string "ETag") tuples, each corresponding to the Effective Request URI (RFC7230, Section 5.5) of a cached response RFC7234 and its entity-tag RFC7232 (if "validators" is true and if the ETag is available; otherwise, </del> <ins> "ETag" a string corresponding to the entity-tag RFC7232 of a cached response RFC7234 (if the ETag is available; otherwise, </ins> null); <del> "P", an integer that MUST be a power of 2 smaller than 2**32, that indicates the probability of a false positive that is acceptable, expressed as "1/P". </del> <ins> "maxcount" - max number of cuckoo hops </ins> <del> "digest-value" can be computed using the following algorithm: </del> <ins> "digest-value" </ins> <del> Let N be the count of "URLs"' members, rounded to the nearest power of 2 smaller than 2**32. </del> <ins> Let "f" be the value of the first byte of "digest-value". </ins> <del> Let "hash-values" be an empty array of integers. </del> <ins> Let "b" be the bucket size, defined as 4. </ins> <del> For each ("URL", "ETag") in "URLs", compute a hash value (hash) and append the result to "hash-values". </del> <ins> Let "N" be the value of the second to fifth bytes of "digest- value" in big endian form. </ins> <del> Sort "hash-values" in ascending order. </del> <ins> Let "key" be the return value of key with "URL" and "ETag" as inputs. </ins> <del> Let "digest-value" be an empty array of bits. </del> <ins> Let "h1" be the return value of hash with "key" and "N" as inputs. </ins> <del> Write log base 2 of "N" to "digest-value" using 5 bits. </del> <ins> Let "fingerprint" be the return value of fingerprint with "key" and "f" as inputs. </ins> <del> Write log base 2 of "P" to "digest-value" using 5 bits. </del> <ins> Let "fingerprint-string" be the value of "fingerprint" in base 10, expressed as a string. </ins> <del> Let "C" be -1. </del> <ins> Let "h2" be the return value of hash with "fingerprint-string" and "N" as inputs, XORed with "h1". </ins> <del> For each "V" in "hash-values": </del> <ins> Let "h" be either "h1" or "h2", picked in random. </ins> <del> If "V" is equal to "C", continue to the next "V". </del> <ins> Let "position_start" be 40 + "h" * "f" * "b". </ins> <del> Let "D" be the result of "V - C - 1". </del> <ins> Let "position_end" be "position_start" + "f" * "b". </ins> <del> Let "Q" be the integer result of "D / P". </del> <ins> While "position_start" < "position_end": </ins> <del> Let "R" be the result of "D modulo P". </del> <ins> Let "bits" be "f" bits from "digest_value" starting at "position_start". </ins> <del> Write "Q" '0' bits to "digest-value". </del> <ins> If "bits" is all zeros, set "bits" to "fingerprint" and terminate these steps. </ins> <del> Write 1 '1' bit to "digest-value". </del> <ins> Add "f" to "position_start". </ins> <del> Write "R" to "digest-value" as binary, using log2("P") bits. </del> <ins> Substract "f" from "position_start". </ins> <del> Let "C" be "V" </del> <ins> Let "fingerprint" be the "f" bits starting at "position_start". </ins> <del> If the length of "digest-value" is not a multiple of 8, pad it with 0s until it is. </del> <ins> Let "h1" be "h" </ins> <del> 2.1.2. </del> <ins> Substract 1 from "maxcount". </ins> <del> Given: </del> <ins> If "maxcount" is zero, return an error. </ins> <del> "URL", an array of characters </del> <ins> Go to step 7. </ins> <del> "ETag", an array of characters </del> <ins> 2.1.3. Given the following inputs: </ins> <del> "validators", a boolean </del> <ins> "URL" a string corresponding to the Effective Request URI (RFC7230, Section 5.5) of a cached response </ins> <del> "N", an integer </del> <ins> "ETag" a string corresponding to the entity-tag RFC7232 of a cached response RFC7234 (if the ETag is available; otherwise, null); </ins> <del> "P", an integer </del> <ins> "digest-value" </ins> <del> "hash-value" can be computed using the following algorithm: </del> <ins> Let "f" be the value of the first byte of "digest-value". Let "b" be the bucket size, defined as 4. Let "N" be the value of the second to fifth bytes of "digest- value" in big endian form. Let "key" be the return value of key with "URL" and "ETag" as inputs. Let "h1" be the return value of hash with "key" and "N" as inputs. Let "fingerprint" be the return value of fingerprint with "key" and "f" as inputs. Let "fingerprint-string" be the value of "fingerprint" in base 10, expressed as a string. Let "h2" be the return value of hash with "fingerprint-string" and "N" as inputs, XORed with "h1". Let "h" be "h1". Let "position_start" be 40 + "h" * "f" * "b". Let "position_end" be "position_start" + "f" * "b". While "position_start" < "position_end": Let "bits" be "f" bits from "digest_value" starting at "position_start". If "bits" is "fingerprint", set "bits" to all zeros and terminate these steps. Add "f" to "position_start". If "h" is not "h2", set "h" to "h2" and return to step 9. 2.1.4. Given the following inputs: "key", an array of characters "f", an integer indicating the number of output bits Let "hash-value" be the SHA-256 message digest RFC6234 of "key", expressed as an integer. Let "h" be the number of bits in "hash-value" Let "fingerprint-value" be 0 While "fingerprint-value" is 0 and "h" > "f": Let "fingerprint-value" be the "f" least significant bits of "hash-value". Let "hash-value" be the "h"-"f" most significant bits of "hash- value". Substract "f" from "h". If "fingerprint-value" is 0, let "fingerprint-value" be 1. Return "fingerprint-value". Note: Step 5 is to handle the extremely unlikely case where a SHA-256 digest of "key" is all zeros.  The implications of it means that there's an infitisimaly larger probability of getting a "fingerprint- value" of 1 compared to all other values.  This is not a problem for any practical purpose. 2.1.5. Given the following inputs: "URL", an array of characters "ETag", an array of characters </ins> Let "key" be "URL" converted to an ASCII string by percent- encoding as appropriate RFC3986. <del> If "validators" is true and "ETag" is not null: </del> <ins> If "ETag" is not null: </ins> Append "ETag" to "key" as an ASCII string, including both the "weak" indicator (if present) and double quotes, as per RFC7232, Section 2.3. <ins> Return "key" 2.1.6. Given the following inputs: "key", an array of characters. "N", an integer "hash-value" can be computed using the following algorithm: </ins> Let "hash-value" be the SHA-256 message digest RFC6234 of "key", <del> expressed as an integer. </del> <ins> truncated to 32 bits, expressed as an integer. </ins> <del> Truncate "hash-value" to log2( "N" * "P" ) bits. </del> <ins> Return "hash-value" modulo N. </ins> 2.2.

Related discussion:
text: I've got an incomplete initial reference implementation at URL It doesn't yet include removal and querying (that's what I'll be adding next), but I did run it on a list of ~3250 URL (which I got out of my main profile chrome://cache/) and it seems to be creating reasonable sized digests. One more advantage, the digests seem to be highly compressible when sparse. Results so far: Digest with 1021 entries (so room for ~4K URLs): 5621 in-memory, 5233 gzipped (when filled with 3250 URLs). Digest with 2503 entries (so room for ~10K URLs): 13772 in-memory, 6879 gzipped (same 3250 URLs). Digest with 7919 entries (so room for ~31K URLs): 43560 in-memory, 9984 gzipped (same 3250 URLs). In practice, I think ~1000 entries is most probably enough, but it's good to know we can increase the digest size (to avoid having to recreate it), without significant over-the-wire penalty.
OK, I now have a complete reference implementation and it seems to be working fine. It also exposed an issue with the initial algorithm, forcing table allocation to accommodate a power of 2 number of entries. Latest results for 3250 URLs taken from my cache: One note: the 1021 entries table had 35 collisions, so it seems like it's insufficient for that number of URLs, unless we're willing to absorb extra pushes for ~1% of the resources.
NAME Interesting! It's good to know that we have numbers now. What is the value of P (the false positive ratio) that you used?
P=8 (so 1/256 false positive)
Note that the numbers here may be possible to further optimize. One example is semi-sorting of the buckets which the Cuckoo-Filters paper mentions, and which I have not yet implemented. It adds some runtime complexity, but can reduce the fingerprint size per resource by a full bit, so could have resulted in ~9% smaller digests in this case.
NAME Awesome! Me not being familiar with these data structures (despite reading Wikipedia article ), why does the 1/256 probability (~4/1000) result in 35 collisions?