jiebi/CodeConvo huggingface dataset:
Change:
This is a follow-up to , removing a few dangling references to the helper state blob.

Discussion:
the reports in sub-batches of 10.  Each sub-batch takes two round- trips to process. <del> In order to allow the helpers to retain minimal state, the helper can attach a state parameter to its response, with the leader returning the state value in the next request, thus offloading the state to the leader.  This state value MUST be cryptographically protected as described in agg-init. </del> [OPEN ISSUE: Should there be an indication of whether a given aggregate request is a continuation of a previous sub-batch?]

Implement some form of batch-to-report index, as well as inter- and intra-batch replay mitigation storage, which includes some way <del> of tracking batch report size with optional support for state offloading.  Some of this state may be used for replay attack mitigation.  The replay mitigation strategy is described in anti- replay. </del> <ins> of tracking batch report size.  Some of this state may be used for replay attack mitigation.  The replay mitigation strategy is described in anti-replay. </ins> Beyond the minimal capabilities required of helpers, leaders are generally required to:

Implement and store state for the form of inter- and intra-batch replay mitigation in anti-replay; and <del> Store helper state. </del> 5.1.3. Collectors statefully interact with aggregators to produce an

schemes like Prio in which the input-validation protocol is only run once per report, each aggregator only needs to store its aggregate share for each possible batch interval, along with the number of <del> times the aggregate share was used in a batch.  (The helper may store its aggregate shares in its encrypted state, thereby offloading this state to the leader.)  This is due to the requirement that the batch interval respect the boundaries defined by the DAP parameters.  (See batch-parameter-validation.) </del> <ins> times the aggregate share was used in a batch.  This is due to the requirement that the batch interval respect the boundaries defined by the DAP parameters.  (See batch-parameter-validation.) </ins> 6.
Change:
This fixes a reference that was pointing at the wrong section. The behavior relating to is under "Input Share Validation".

Discussion:
given task.  If the leader receives a report from a client whose nonce is in this set, it simply ignores it.  A helper who receives an encrypted input share whose nonce is in this set replies to the <del> leader with an error as described in collect-flow. </del> <ins> leader with an error as described in input-share-batch-validation. </ins> [OPEN ISSUE: This has the potential to require aggreagtors to store nonce sests indefinitely.  See issue#180.]
Change:
(opinionatedly: please see discussion on that issue)
Currently, DAP uses the following media types (defined in section 7.1): Note that the first uses the top-level type, while the remainder use the top-level type. For consistency, I suggest we use the same top-level type for all messages. Which type is more appropriate? Per : is for "either uninterpreted binary data or information to be processed by an application" is for "an encapsulated message"; all of the motivating examples (see section 5.2) are around encaspulating one message format inside another, or fragmenting a message into multiple chunks. provides further guidance that "multipart and message are composite types; that is, they provide a means of encapsulating zero or more objects, each one a separate media type." Given that DAP messages are not a method of encapsulating a message of a different type, I suggest that is the more appropriate top-level type to use. This is also supported by the rather large number of media types vs the relatively tiny number of media types in . (this issue fell out of discussion on URL)
Agreed, seems better.
The types were chosen in PR on a tentative basis, and marked with "OPEN ISSUE: Solicit review of these allocations from domain experts." Based on the encapsulation reasoning, I agree that be most appropriate for our media types.
As a non-HTTP expert, I agree with the rationale. Thanks, NAME based on the text quoted by NAME in the linked issue. It would be great to get feedback from folks who have more experience working on HTTP-based applications.

Discussion:
HpkeConfig hpke-config: "application/dap-hpke-config" <del> Report upload-request: "message/dap-report" </del> <ins> Report upload-request: "application/dap-report" </ins> <del> AggregateInitializeReq collect-flow: "message/dap-aggregate- </del> <ins> AggregateInitializeReq collect-flow: "application/dap-aggregate- </ins> initialize-req" <del> AggregateInitializeResp collect-flow: "message/dap-aggregate- </del> <ins> AggregateInitializeResp collect-flow: "application/dap-aggregate- </ins> initialize-resp" <del> AggregateContinueReq collect-flow: "message/dap-aggregate- </del> <ins> AggregateContinueReq collect-flow: "application/dap-aggregate- </ins> continue-req" <del> AggregateContinueResp collect-flow: "message/dap-aggregate- </del> <ins> AggregateContinueResp collect-flow: "application/dap-aggregate- </ins> continue-resp" <del> AggregateShareReq collect-flow: "message/dap-aggregate-share-req" </del> <ins> AggregateShareReq collect-flow: "application/dap-aggregate-share- req" </ins> <del> AggregateShareResp collect-flow: "message/dap-aggregate-share- </del> <ins> AggregateShareResp collect-flow: "application/dap-aggregate-share- </ins> resp" <del> CollectReq collect-flow: "message/dap-collect-req" </del> <ins> CollectReq collect-flow: "application/dap-collect-req" </ins> <del> CollectResp collect-flow: "message/dap-collect-resp" </del> <ins> CollectResp collect-flow: "application/dap-collect-resp" </ins> The definition for each media type is in the following subsections.
Change:
Previously, report-replayed would be returned if the same client report was aggregated more than once. However, some VDAFs (e.g. poplar1) will require repeatedly aggregating a given client report with varying aggregation parameters. This commit relaxes the condition on which report-replayed will be returned to allow aggregating the same client report multiple times, as long as each aggregation has a distinct aggregation parameter.
This is odd because the aggregation parameter is opaque and it's not clear if any change to it should allow aggregating a report again. For example, if aggregation is allowed only once for privacy reasons but some kind of shared random number is needed as agg_param this will be an issue. I think checking if another aggregation is allowed has to be done depending on the VDAF and the task.
There is a separate check, under the section "Validating Batch Parameters", that each report is used in at most aggregate shares. This should be set to 1 for Prio3 and VDAFs like you describe. If for a task is 1, then the present change will make no difference, but if for a task is greater than one, we need this change in order to allow that second batch.
Currently, the report share error is returned "if the report has already been aggregated" (). Some VDAFs, like , will require repeatedly aggregating the same client reports with a varying aggregation parameter. I think the specification language would effectively block this use-case currently. I think the fix is to return a error if the report has already been aggregated with the same aggregation parameter. Note that this change would not allow a malicious leader to potentially aggregate a different set of reports with a different aggregation parameter: the existing error would be returned if a client attempted to upload another report after the first collection attempt. (Given the small size of the proposed fix, I'll send a PR shortly, but discussion on this issue/the WG mailing list takes precedence -- I'll update the PR to follow any changes from discussion.)
Thanks NAME -- this change LGTM.

Discussion:
prevents additional reports from being aggregated after its batch has already been collected. <del> Check if the report has already been aggregated.  If this check fails, the input share is marked as invalid with the error "report-replayed".  This is the case if the report was used in a previous aggregate request and is therefore a replay.  An aggregator may also choose to mark an input share as invalid with the error "report-dropped" under the conditions prescribed in anti-replay. </del> <ins> Check if the report has already been aggregated with this aggregation parameter.  If this check fails, the input share is marked as invalid with the error "report-replayed".  This is the case if the report was used in a previous aggregate request and is therefore a replay.  An aggregator may also choose to mark an input share as invalid with the error "report-dropped" under the conditions prescribed in anti-replay. </ins> If both checks succeed, the input share is not marked as invalid.
Change:
The next VDAF draft is out. It includes some minor changes that we'll need to adopt here. URL

Discussion:
(described in hpke-config); see compliance for information about the HPKE configuration algorithms. <del> "vdaf_verify_param": The aggregator's VDAF verification parameter output by the setup algorithm computed jointly by the aggregators before the start of the DAP protocol VDAF).  [OPEN ISSUE: This is yet to be specified.  See issue#161.] </del> <ins> "vdaf_verify_key": The VDAF verification key shared by the aggregators.  This key is used in the aggregation sub-protocol (aggregate-flow).  [OPEN ISSUE: The manner in which this key is distributed may be relevant to the VDAF's security.  See issue#161.] </ins> The helper stores a bearer token used to authenticate HTTP requests from the leader.  Likewise, the leader stores a bearer token to

Each aggregator runs this procedure for a given input share with corresponding nonce as follows: <del> "vdaf_verify_param" is the public VDAF parameter, and "agg_param" is the opaque aggregation parameter.  If either step fails, the aggregator marks the report as invalid with error "vdaf-prep-error". </del> <ins> "vdaf_verify_key" is the VDAF verification key shared by the aggregators; "agg_id" is the aggregator ID ("0x00" for the Leader and "0x01" for the helper); and "agg_param" is the opaque aggregation parameter distributed to the aggregtors by the collector. </ins> <del> Otherwise, the value "out" is interpreted as follows.  If this is the last round of the VDAF, then "out" is the aggregator's output share. Otherwise, "out" is the pair "(prep_state, prep_msg)". </del> <ins> If either step fails, the aggregator marks the report as invalid with error "vdaf-prep-error".  Otherwise, the value "out" is interpreted as follows.  If this is the last round of the VDAF, then "out" is the aggregator's output share.  Otherwise, "out" is the pair "(prep_state, prep_msg)". </ins> 4.3.2.

each share in the candidate report set until the underlying VDAF moves into a terminal state, yielding an output share for all aggregators or an error.  This phase may involve multiple rounds of <del> interaction depending on the underlying VDAF parameters.  Each round trip is initiated by the leader. </del> <ins> interaction depending on the underlying VDAF.  Each round trip is initiated by the leader. </ins> 4.3.2.1.
Change:
This adds a field to collect responses with the number of reports that contributed to the aggregation. Collectors will likely want this count to aid in the interpretation of aggregate results. (see ) There is also a change in draft-irtf-cfrg-vdaf-02 that opens the door for VDAFs that would require the report count during unsharding. (see cfrg/draft-irtf-cfrg-vdaf) Note that while this field is not part of the adjacent s, it will be protected by to the confidentiality of HTTPS and the leader-collector bearer token authentication.
I'm open to pushing it into the ciphertext, but my thinking was that a tampered reportcount would only affect robustness, not privacy, and thus we only need to arrive at the correct reportcount when all aggregators follow the protocol.
NAME can you please rebase? Sorry!
Collectors will want to know how many inputs are included in the aggregations they receive, for a number of reasons: they may wish to compute a mean over a privately aggregated sum, or track metrics on how many clients are providing inputs. struct should include a count of how many contributions went into an aggregation.
The linked PR only added a to the structure, which communicates it to the helper. I'm reopening this issue because the collector does not yet receive this information. In addition to the statistics-related motivation above, a recent PR to the VDAF spec, cfrg/draft-irtf-cfrg-vdaf, expects the number of reports to be available when unsharding.
This was completed in .
OK, seems reasonable to me.

Discussion:
HTTP GET requests to the collect job's URI with HTTP status code 200 OK and a body consisting of a "CollectResp": <del> The "encrypted_agg_shares" field is the vector of encrypted aggregate shares.  They MUST appear in the same order as the aggregator endpoints list of the task parameters. </del> <ins> "report_count" is the number of reports included in the aggregation. "encrypted_agg_shares" is the vector of encrypted aggregate shares.  They MUST appear in the same order as the aggregator endpoints list of the task parameters. </ins> If obtaining aggregate shares fails, then the leader responds to subsequent HTTP GET requests to the collect job URI with an HTTP
Change:
NAME let's squash-then-merge this change.
contains a nice definition and discussion of the notions of privacy (meaning protecting any individual client's measurement from being seen) and robustness (meaning that the aggregates delivered to a collector are accurate aggregations over the inputs). DAP's security considerations has definitions of privacy and correctness, and we refer to robustness and soundness in a few places. We should align our definitions with VDAF (or just refer to VDAF's definitions) and consistently use the term "robustness" throughout DAP.
PR, please!
I'm going to get this done for draft-02, hopefully early next week.

Discussion:
coerce an aggregator into diverting from the protocol specified (e.g., by divulging its input shares to the attacker). <del> In the presence of this adversary, DAP aims to achieve the following high-level secure aggregation goals: Privacy.  Clients trust that some aggregator is honest.  That is, as long as at least one aggregator executes the protocol faithfully, the parties learn nothing beyond the aggregate result (i.e., the output of the aggregation function computed over the honest measurements). Correctness.  The collector trusts that the aggregators execute the protocol correctly.  That is, as long as the aggregators execute the protocol faithfully, a malicious client can skew the aggregate result only by reporting a false (untruthful) measurement.  The result cannot be influenced in any other way. </del> <ins> In the presence of this adversary, DAP aims to achieve the privacy and robustness security goals described in VDAF's Security Considerations section. </ins> Currently, the specification does not achieve these goals.  In particular, there are several open issues that need to be addressed

[[OPEN ISSUE: The threat model for Prio -- as it's described in the original paper and BBCGGI19 -- considers <del> a malicious client (attacking soundness) </del> <ins> a malicious client (attacking robustness) </ins> a malicious subset of aggregators (attacking privacy).  In <del> particular, soundness isn't guaranteed if any one of the aggregators </del> <ins> particular, robustness isn't guaranteed if any one of the aggregators </ins> is malicious; in theory it may be possible for a malicious client and <del> aggregator to collude and break soundness.  Is this a contingency we </del> <ins> aggregator to collude and break robustness.  Is this a contingency we </ins> need to address?  There are techniques in BBCGGI19 that account for this; we need to figure out if they're practical.]]
Change:
I-D Nits complains of outdated reference for: ** Obsolete normative reference: RFC 7234 (Obsoleted by RFC 9111) We should update tit.
Thanks!

Discussion:
Aggregators can control this cached lifetime with the Cache-Control header, as follows: <del> Clients SHOULD follow the usual HTTP caching RFC7234 semantics for </del> <ins> Clients SHOULD follow the usual HTTP caching RFC9111 semantics for </ins> key configurations. Note: Long cache lifetimes may result in clients using stale HPKE
Change:
NAME I think we should wait for one of NAME or NAME before merging.
s4.1 includes the following: Future specifications can introduce new query types as needed. That means this I-D needs an IANA consideration to address registering a query type.

Discussion:
The time-interval query type is described in time-interval-query; the fixed-size query type is described in fixed-size-query.  Future specifications can introduce new query types as needed (see query- <del> type-reg.  A query includes parameters used by the Aggregators to </del> <ins> type-reg).  A query includes parameters used by the Aggregators to </ins> select a batch of reports specific to the given query type.  A query is defined as follows:
Change:
cc/ NAME
The presentation syntax is: opaque public_share; Are there ever going to be no public shares in a Report?
EDITED: Yes, Prio3 has an empty public share. This is mainly needed for Poplar1. From the :
NAME if you want, we can add text that makes sure your question is answered. Otherwise I don't think there's any action to take here.
I think the answer should be: "Yes, Prio3 doesn't need public shares."
Thanks for the update.

Discussion:
This message is called the Client's report.  It consists of the task ID, report metadata, the "public share" output by the VDAF's input- distribution algorithm, and the encrypted input share of each of the <del> Aggregators.  The metadata consists of the following fields: </del> <ins> Aggregators.  (Note that the public share might be empty, depending on the VDAF.  For example, Prio3 has an empty public share, but Poplar1 does not.  See VDAF.)  The header consists of the task ID and report "metadata".  The metadata consists of the following fields: </ins> A timestamp representing the time at which the report was generated.  Specifically, the "time" field is set to the number of
Change:
The prepare_shares fields were always of type PrepareStep. To match the convention used for most other field names, I change the name of the field to match its type. (There is no "PrepareShare" type.)

Discussion:
"(prep_state, prep_msg)". To carry out this step, for each PrepareStep in <del> AggregateContinueReq.prepare_shares received from the leader, the </del> <ins> AggregateContinueReq.prepare_steps received from the leader, the </ins> helper performs the following check to determine if the report share should continue being prepared:

"prep_msg" is its next VDAF message. This output message for each report in <del> AggregateContinueReq.prepare_shares is then sent to the leader in an </del> <ins> AggregateContinueReq.prepare_steps is then sent to the leader in an </ins> AggregateContinueResp message, structured as follows: <del> The order of AggregateContinueResp.prepare_shares MUST match that of the PrepareStep values in "AggregateContinueReq.prepare_shares".  The </del> <ins> The order of AggregateContinueResp.prepare_steps MUST match that of the PrepareStep values in "AggregateContinueReq.prepare_steps".  The </ins> helper's response to the leader is an HTTP status code 200 OK whose body is the AggregateContinueResp and media type is "message/dap- aggregate-continue-resp".  The helper then awaits the next message
Change:
The Aggregate Share Encryption section still had a few references to "batch interval", but the encryption process now uses the "batch selector".

Discussion:
The collector decrypts these aggregate shares using the opposite process.  Specifically, given an encrypted input share, denoted <del> "enc_share", for a given batch interval, denoted "batch_interval", </del> <ins> "enc_share", for a given batch selector, denoted "batch_selector", </ins> decryption works as follows: where "sk" is the HPKE secret key, "task_id" is the task ID for the
Change:
NAME please have another look
This seems useful, since it disambiguates "tasklifetime" and "batchlifetime". The former, as i understand it, is a timestamp; and the latter is a number of queries. cc/ NAME for thoughtful review here

Discussion:
too many times.  This is determined by the maximum number of times a batch can be queried, "max_batch_query_count".  Unless the query has been issued less than "max_batch_query_count" times, the Aggregator <del> MUST abort with error of type "batchLifetimeExceeded". </del> <ins> MUST abort with error of type "batchQueriedTooManyTimes". </ins> Finally, the Aggregator checks that the batch does not contain a report that was included in any previous batch.  If this batch
Change:
In IANA all media types seem to be of "application" type instead of "message" type, this PR is to align the mismatch in the protocol description and adapt to the media types specified in IANA.
Nice catch!

Discussion:
Let "[aggregator]" denote the helper's API endpoint.  The leader sends a POST request to "[aggregator]/aggregate" with its AggregateInitializeReq message as the payload.  The media type is <del> "message/dap-aggregate-initialize-req". </del> <ins> "application/dap-aggregate-initialize-req". </ins> 4.4.1.2.

is finished for the report share. The helper's response to the leader is an HTTP status code 200 OK <del> whose body is the AggregateInitializeResp and media type is "message/ dap-aggregate-initialize-resp". </del> <ins> whose body is the AggregateInitializeResp and media type is "application/dap-aggregate-initialize-resp". </ins> Upon receipt of a helper's AggregateInitializeResp message, the leader checks that the sequence of PrepareStep messages corresponds

"AggregateContinueReq.task_id"'s parameters except its own, the leader sends a POST request to "[aggregator]/aggregate" with AggregateContinueReq as the payload and the media type set to <del> "message/dap-aggregate-continue-req". </del> <ins> "application/dap-aggregate-continue-req". </ins> 4.4.2.2.

The order of AggregateContinueResp.prepare_steps MUST match that of the PrepareStep values in "AggregateContinueReq.prepare_steps".  The helper's response to the leader is an HTTP status code 200 OK whose <del> body is the AggregateContinueResp and media type is "message/dap- </del> <ins> body is the AggregateContinueResp and media type is "application/dap- </ins> aggregate-continue-resp".  The helper then awaits the next message from the leader.