Draft: draft-ietf-ppm-dap

Commentary: Since aggregators already need to store nonce sets for the purpose of replay prevention , it does n't make much sense to also offload part of their state to the leader . Instead , aggregators might benefit from a unique index into some local storage for keeping track of per - aggregation job state . This change drops the helper_state in favor of such an index . 
 A few points : seems strictly less flexible . The helpers could already store an index if they wanted , but now they are required to do so . ' t you store nonce sets in the helper state ? 
 Yeah , they could store an index in helperstate . However , helperstate is a " per - aggregation " storage mechanism , not a " per - task " mechanism , and ( I think ) the replay state needs to cover the entire task . So we could ( a ) change helper_state to be a per - task thing , in which case it 's not clear 2 ^ 16 is enough space , or if we want to be paying that bandwidth for the duration of the task , or ( b ) go with the layer of indirection as in here . On balance , I think ( b ) is probably better ? 
 This can get really hairy . Think of this way : Storing nonce sets is required for report replay protection , which is needed for privacy . Depending on what state the helper has locally for managing the " version " of the state blob , it may be possible for the leader to " replay " an old state and bypass replay protection for reports . There are other ways that helper state gets hairy . For example , we have n't been precise yet about how handling concurrent aggregation flows : If the helper state is " per - task " , then concurrent flows would be a problem . Overall the goal of this change is to remove this complexity altogether . We can always add it back later if we need to . 
 I do n't agree that this change reduces complexity . AFAICT , this is two changes : has the leader assign the ID . restricts the ID space to uint32 . ISTM that it 's just as straightforward for the helper to assign the i d at the start of the job , and then use any format it chooses , which the current system allows . I get that you think that certain ways of assigning and using that ID are complex , but that 's not our problem because the helper can figure it out for themselves . 
 You mean 32 - byte strings , right ?  is just 8 bytes , whereas the AggregationJobID is 32 bytes . Anther important thing it does is remove the state blob from the AggregateShareReq . Here 's the bit I think you 're missing : Right now we require the leader to send the same it received in the helper 's previous response . This means that all aggregate requests , i.e. , AggregateInitReq , AggregateContReq , and AggregateShareReq , need to be issued one after the other . In particular , it 's not possible to run multiple aggregation flows concurrently . You 're forced to do before you do ( Note that the " aggregation flow " excludes the AggregateShareReq , which is used to combine the results across multiple aggregation flows . ) It 's going to be important for scalability that we can run multiple aggregation flows simultaneously . That means that we need , at a minimum , the following things : ' t carry state across aggregation flows sure there 's a way for a helper to " link " aggregate requests pertaining to the same aggregation flow This PR accomplishes both of these , but as you point out , is more strict than necessary . Perhaps we could keep the helper state blob as - is , but remove it from the AggregateShareReq and mandate only that the helper state is maintained for the duration of a single aggregation flow . 
 I 'm undecided about whether this is a good or bad change , but it should be decided on its own merits , not buried in a PR entitled " Replace helperstate with index into per - aggregator storage " 
 I believe this change does yield a significant reduction in complexity , because we no longer have to spell out how the helper needs to protect its state from the leader . Aggregation job IDs do not need to be kept secret from the leader , nor does the helper need to need to implement any anti - replay protections for them ( because all the sensitive state is now assumed to be in the helper 's database ) . 
 The reasoning for this was originally discussed on issue . This issue did n't get much attention , so we did n't expect the change to be controversial . We certainly did n't intend to bury it . We tried implementing both and thought this approach solved several problems compared to the status quo . Would you prefer that we take this too the list ? 
 Let 's just move this discussion to the list 
 I 've just read that issue several times and I do n't really see much discussion of issuing aggregation requests in parallel . The question here is not primarily about the list versus not the list . It 's about the process we are going to follow for landing changes . In general , substantive changes should be getting the consensus of the WG prior to being landed . At this relatively early phase , IETF sometimes allows editors some latitude to make design decisions ahead of time that will then be reviewed by the group when the next draft is published . However , typically the way to do this would be to email the list with a clear explanation of what the changes were and a notification that you were going to land a set of PRs unless someone objected by ( say ) a week away . 
 mind rebasing to fix conflicts ? 
 done . 
 and I were debating the . Now that helpers have to store all the report nonces and are also likely to be expected to store their own report shares , I argue that we should get rid of it : we do n't need it and it entails a bunch of complicated security considerations . Helpers need to encrypt their state so leaders ca n't see it , need to protect themselves against leaders replaying old state blobs and they need to version it so that a helper can deal with a blob constructed by an older version of itself .  ' rebuttal is that even if you do n't use the blob to store something like a set of nonces or a counter or a partial aggregate share , it 's still useful as a key into the storage where the helper keeps its state . This is helpful for a helper running on something like a Cloudflare Worker , where ephemeral servers are launched in response to incoming messages , and so a different helper instance might handle the sequence of and requests . We agreed that we do n't need an arbitrarily large opaque blob shipped back and forth between leader and helper to solve that problem . All you need is a unique value like a UUID so that a stateless helper can recover context from storage . This will let us delete a whole bunch of SHOULD and MAY discussion from the spec . 
 I think we also owe a definition of an " aggregation job " , which is evolving in 
 Some discussion of the implications of helper state to parallelism is in . 
 The changes LGTM , but of course we should take this to the ppm list before merging .
Paragraph (version A):
Once a set of clients have uploaded their reports to the leader, the
leader can send them to the helpers to be verified and aggregated.
In order to enable the system to handle very large batches of
reports, this process can be performed incrementally.  To aggregate a
set of reports, the leader sends a message to each helper containing
those report shares.  The helper then processes them (verifying the
proofs and incorporating their values into the ongoing aggregate) and
replies to the leader.
Paragraph (version B):
Once a set of clients have uploaded their reports to the leader, the
leader can send them to the helpers to be verified and aggregated.
In order to enable the system to handle very large batches of
reports, this process can be performed incrementally.  Verification
of a set of reports is referred to as an aggregation job.  Each
aggregation job is associated with exactly one PPM task, and a PPM
task can have many aggregation jobs.  Each job is associated with an
ID that is unique within the context of a PPM task in order to
distinguish different jobs from one another.  Each aggregator uses
this ID as an index into per-job storage, e.g., to keep track of
report shares that belong to a given aggregation job.
Paragraph (version B):
To run an aggregation job, the leader sends a message to each helper
containing the report shares in the job.  The helper then processes
them (verifying the proofs and incorporating their values into the
ongoing aggregate) and replies to the leader.
Paragraph (version B):
Generate a fresh AggregationJobID.  This ID MUST be unique within
the context of the corresponding PPM task.  It is RECOMMENDED that
this be set to a random string output by a cryptographically
secure pseudorandom number generator.
Paragraph (version A):
The "nonce" and "extensions" fields of each ReportShare match that in
the Report uploaded by the client.  The "encrypted_input_share" field
is the "HpkeCiphertext" whose index in
"Report.encrypted_input_shares" is equal to the index of the
aggregator in the task's "aggregator_endpoints" to which the
AggregateInitReq is being sent.  The "agg_param" field is an opaque,
VDAF-specific aggregation parameter.  The "helper_state" parameter
contains the helper's state.  This is an optional parameter of an
aggregate request that the helper can use to carry state across
requests and across aggregate flows.
Paragraph (version B):
The "nonce" and "extensions" fields of each ReportShare match that in
the Report uploaded by the client.  The "encrypted_input_share" field
is the "HpkeCiphertext" whose index in
"Report.encrypted_input_shares" is equal to the index of the
aggregator in the task's "aggregator_endpoints" to which the
AggregateInitReq is being sent.  The "agg_param" field is an opaque,
VDAF-specific aggregation parameter.  The "job_id" parameter contains
the leader's chosen AggregationJobID.
Paragraph (version A):
The "helper_state" parameter contains the helper's initial state.
This is an optional parameter of an aggregate request that the helper
can use to carry state across requests.  At least part of the state
will usually need to be encrypted in order to protect user privacy.
However, the details of precisely how the state is encrypted and the
information that it carries is up to the helper implementation.
Paragraph (version A):
The rest of the message is a sequence of PrepareStep values, the
order of which matches that of the ReportShare values in
"AggregateInitReq.report_shares".  Each report that was marked as
invalid is assigned the PrepareStepResult "failed".  Otherwise, the
PrepareStep is either marked as continued with the output "prep_msg",
or is marked as finished if the VDAF preparation process is finished
for the report share.
Paragraph (version B):
The "job_id" parameter is equal to AggregateInitReq.job_id.  The rest
of the message is a sequence of PrepareStep values, the order of
which matches that of the ReportShare values in
"AggregateInitReq.report_shares".  Each report that was marked as
invalid is assigned the PrepareStepResult "failed".  Otherwise, the
PrepareStep is either marked as continued with the output "prep_msg",
or is marked as finished if the VDAF preparation process is finished
for the report share.
Paragraph (version A):
"helper_state" is the helper's state, which is carried across
requests from the leader.
Paragraph (version B):
"job_id" is the ID of the aggregation job.