Change:
Having to track all report IDs is a burden for sharding anti-replay storage across time. This change clarifies that, while tracking report IDs is sufficient, it is also sufficient to track any report metadata that contains the reprot ID.
cc/ NAME
LGTM
Squashed
Anti-replay is enforced in . Both DAP-01 and DAP-02 read: What's not fully specified, at least in this section, is what it means for a report to "have been aggregated (with this aggregation parameter)". For that we have been referring to the anti-replay section. reads: The report ID is defined as On the other hand, reads: The nonce is defined as Thus in DAP-01 we are required to check for collisions of the timestamp+random value, whereas in DAP-02 we are required to check for collisions of just a random value (report ID). The latter is stricter, since it disallows repeated random values. After discussing some implementation stuff with NAME I realized that this may have been an unintended regression. From a security standpoint, I think the DAP-02 requirement is better, since it forces the client implementation to generate a fresh report ID each time it generates a report. Otherwise it is easier to fingerprint reports from the same client. Does this violate assumptions in anyone's implementation? Where I can imagine this being problematic is if your implementation currently shards nonce sets into time windows. does this, but we have to amend this behavior to implement fixed-size queries anyway. Reply to this thread if this causes an issue for you and you think we should reconsider this requirement.
From Janus' perspective, I think we are OK with this change. Janus also breaks report sets into time windows -- I don't think this change will affect Janus' implementation of that functionality, though as you note this sharding behavior will change with tasks.
I'm nearly done with the implemenation ... once the code is reviewed and landed I'll try to remember to write up how I ended up organizing data.
Seems fine to me. I can't think of a reason why the client would want to reuse a reportID with a different time for the same task.
I am in favor of recommending that clients generate a good random id for every report AND for allowing the timestamp to be considered as part of the anti-replay strategy. While an unbounded memory of seen report ids would indeed be excellent protection, in practice this requires unbounded storage of report ids, which is a practical problem for implementations. Replay of a client report can leak information, so we want to assure that a report is aggregated at most once. A report has a report id and a timestamp, both generated by the reporting client. Both are included in the authenticated additional data, and cannot be altered without the tampering being evident. For the rest of this note, we will assume that the client is doing its part for its own privacy by choosing a new random report id for every report, and by having a clock is not hugely inaccurate. Even if the clients are doing their part, we must still be concerned with the actions of other entities, such as a compromised leader, helper, or collector. We also need to prevent third party manipulation, e.g. by replaying captured reports. The use of the timestamp can help us here, because the timestamp cannot be altered by any of these parties without detection. If at least one aggregator is honest and imposes the policy below, then we can use the timestamp to ease our storage burdens. Let’s consider the timestamp in more detail. For reporting accuracy we don’t want to permit just any timestamp to be aggregated. At any given time T by the server’s clock, there are bounds we want to apply based on the timestamp. Reports earlier than some time MinTime = T - MinDelta should be rejected as too old, and reports farther in the future than MaxTime = T + MaxDelta should be rejected for being so confused about the time. We allow reports slightly in the future as we want to allow for clock-skew. Call the closed interval [MinTime, MaxTime] for time T Allowed(T). We will discard reports outside of this interval when we receive them, and also when we process them. We impose the “on reception” policy to avoid queuing work that we’d never do, and we impose on processing to prevent backlog and to allow the definition of LeastRetainedTime (below). Assume a monotonically non-decreasing system clock. Whenever an aggregation begins, we durably record the current MinTime and associate it with the active state for the aggregation job, using appropriate locking in concurrent implementations. Define the LeastRetainedTime as the minimum of the current MinTime and the recorded MinTime over all active aggregation jobs (if any). It is impossible to aggregate any report before this time, as it will be rejected in any new job and isn’t present in any existing job. Garbage collection may safely remove any recorded report metadata with a timestamp before the LeastRetainedTime. We now only have to remember report ids that are greater than or equal to the LeastRetainedTime. If clients choose random ids with every report submitted, and there is at least one honest aggregator, then any given report cannot be replayed. Note that in cases where a server is so flooded with reports that it cannot process or remember them, then the DAP spec still has the "report_dropped" error available to ensure that reports are not aggregated when anti-replay cannot be assured.
Thanks NAME Full disclosure: Bob is helping out with Daphne development. When he showed me this analysis, I was pretty convinced that relaxing this requirement would be safe and worth it to reduce burden on implementations. Although I'm not strongly opposed to keeping the stronger requirement if folks think it is useful. I think the only change that would need to be made is to :
I also agree with Bob's analysis. I think we should make the change described by cjpatton in DAP-03.
This argument is good. One point: while it certainly justifies including the timestamp in the overall anti-replay strategy, it does not necessarily justify checking rather than only for the ID-based check. That is, the overall strategy could be: Check the timestamp; if it's out of the acceptable window, fail due to too old/too new. Check if [or ] is in the set of received reports; if it is, fail due to replayed-report. I would have a preference for doing the ID-based check on only, since the report ID is the primary identifier of the report, but plausibly implementations may want to check timestamps as well (for example, if they bucket report IDs by timestamp). Perhaps we could phrase things such that either implementation is allowed? In practice, as long as the report IDs are properly generated as random values, the timestamp will be irrelevant to the ID-based check. If the leader/helper disagree on whether to check or , the more restrictive check will be the one that is effectively in place. edit: another way of looking at this argument -- for the ID-based replay check, what justification is there for checking over, say, or ?
I think if we assume that honest clients always choose their report ID at random, then any anti-replay check that includes the report ID ought to be sufficient. I agree we could write this in a way that makes this implementation-specific perhaps suggesting that (report_id, timestamp) is one particularly useful relaxation.

Discussion:
report "metadata".  The metadata consists of the following fields: A report ID used by the Aggregators to ensure the report appears <del> in at most one batch.  (See anti-replay.)  The Client MUST generate this by generating 16 random bytes using a </del> <ins> in at most one batch.  (See input-share-validation.)  The Client MUST generate this by generating 16 random bytes using a </ins> cryptographically secure random number generator. A timestamp representing the time at which the report was

not succeed, clients MUST abort and discontinue retrying. The Leader MUST ignore any report pertaining to a batch that has <del> already been collected.  (See early-input-share-validation for details.)  Otherwise, comparing the aggregate result to the previous aggregate result may result in a privacy violation.  (Note that the Helpers enforce this as well.)  The Leader MAY ignore any reports whose timestamp is past the task's "task_expiration".  When it does so, the leader SHOULD abort the upload protocol and alert the client with error "reportTooLate".  Client MAY choose to opt out of the task if its own clock has passed "task_expiration". </del> <ins> already been collected.  (See input-share-validation for details.) Otherwise, comparing the aggregate result to the previous aggregate result may result in a privacy violation.  (Note that the Helpers enforce this as well.)  The Leader MAY ignore any reports whose timestamp is past the task's "task_expiration".  When it does so, the leader SHOULD abort the upload protocol and alert the client with error "reportTooLate".  Client MAY choose to opt out of the task if its own clock has passed "task_expiration". </ins> Leaders can buffer reports while waiting to aggregate them.  The leader SHOULD NOT accept reports whose timestamps are too far in the

Extensions are mandatory-to-implement: If an Aggregator receives a Report containing an extension it does not recognize, then it MUST <del> reject the Report.  (See early-input-share-validation for details.) </del> <ins> reject the Report.  (See input-share-validation for details.) </ins> 4.3.4.

input-share-decryption. Check that the resulting input share is valid as described in <del> early-input-share-validation. </del> <ins> input-share-validation. </ins> Initialize VDAF preparation as described in input-share-prep.

input-share-decryption. Check that the resulting input share is valid as described in <del> early-input-share-validation. </del> <ins> input-share-validation. </ins> Initialize VDAF preparation and initial outputs as described in input-share-prep. <del> [[OPEN ISSUE: consider moving the helper report ID check into early- input-share-validation]] </del> <ins> [[OPEN ISSUE: consider moving the helper report ID check into input- share-validation]] </ins> Once the helper has processed each valid report share in "AggregateInitializeReq.report_shares", the helper then creates an

Before beginning the preparation step, Aggregators are required to perform the following generic checks. <del> Check if the report has never been aggregated but is contained by a batch that has been collected.  If this check fails, the input share MUST be marked as invalid with the error "batch_collected". This prevents additional reports from being aggregated after its batch has already been collected. </del> Check if the report has already been aggregated with this aggregation parameter.  If this check fails, the input share MUST be marked as invalid with the error "report_replayed".  This is the case if the report was used in a previous aggregate request and is therefore a replay. <ins> Implementation note: To detect replay attacks, each Aggregator is required to keep track of the set of reports it has processed for a given task.  Because honest Clients choose the report ID at random, it is sufficient to store the set of IDs of processed reports.  However, implementations may find it helpful to track additional information, like the timestamp, so that the storage used for anti-replay can be sharded efficiently. Check if the report has never been aggregated but is contained by a batch that has been collected.  If this check fails, the input share MUST be marked as invalid with the error "batch_collected". This prevents additional reports from being aggregated after its batch has already been collected. </ins> Depending on the query type for the task, additional checks may be applicable:

Aggregator checks that "len(X) >= min_batch_size" and "len(X) <= max_batch_size", where "X" is the set of reports in the batch. <del> 4.5.7. Using a client-provided report multiple times within a single batch, or using the same report in multiple batches, may allow a server to learn information about the client's measurement, violating the privacy goal of DAP.  To prevent such replay attacks, this specification requires the aggregators to detect and filter out replayed reports. To detect replay attacks, each aggregator keeps track of the set of report IDs pertaining to reports that were previously aggregated for a given task.  If the leader receives a report from a client whose report ID is in this set, it either ignores it or aborts the upload sub-protocol as described in upload-flow.  A Helper who receives an encrypted input share whose report ID is in this set rejects the report as described in early-input-share-validation. [OPEN ISSUE: This has the potential to require aggregators to store report ID sets indefinitely.  See issue#180.] A malicious aggregator may attempt to force a replay by replacing the report ID generated by the client with a report ID its peer has not yet seen.  To prevent this, clients incorporate the report ID into the AAD for HPKE encryption, ensuring that the output share is only recovered if the aggregator is given the correct report ID.  (See upload-request.) Aggregators prevent the same report from being used in multiple batches (except as required by the protocol) by only responding to valid collect requests, as described in batch-validation. </del> 5. The DAP protocol has inherent constraints derived from the tradeoff

and intra-batch replay mitigation storage, which includes some way of tracking batch report size.  Some of this state may be used for replay attack mitigation.  The replay mitigation strategy is <del> described in anti-replay. </del> <ins> described in input-share-validation. </ins> Beyond the minimal capabilities required of helpers, leaders are generally required to:

In addition, for each DAP task, leaders are required to: Implement and store state for the form of inter- and intra-batch <del> replay mitigation in anti-replay. </del> <ins> replay mitigation in input-share-validation. </ins> 5.1.3.

report upload rate, this can result in high storage costs.  To alleviate this burden, DAP allows Aggregators to drop this state as needed, so long as reports are dropped properly as described in <del> early-input-share-validation.  Aggregators SHOULD take steps to mitigate the risk of dropping reports (e.g., by evicting the oldest data first). </del> <ins> input-share-validation.  Aggregators SHOULD take steps to mitigate the risk of dropping reports (e.g., by evicting the oldest data first). </ins> Furthermore, the aggregators must store data related to a task as long as the current time has not passed this task's