Draft: draft-ppm-protocol

Commentary: Modify , and to accommodate more than one helper . Note that the PPM protocol does n't actually support multiple helpers to enhance privacy ( ) but we nonetheless want to define the PDUs so it is possible to introduce that feature in the future . We do this by introducing an type into various struct definitions , allowing participants to tell which aggregator an input share or output share corresponds to . We also introduce a special , reserved value of that allows clients to tell which aggregator is the leader to which reports should be uploaded . Finally , we use the aggregator ID in HPKE context construction instead of the existing constants 0x00 and 0x01 for leader and helper , respectively . Along the way , we introduce the type alias used in few different places . 
 Note that this change is stacked on ( sorry ) . [ x ] make sure this PR targets before merging 
 I merged , so we can retarget this one now . 
 Thanks ! This is retargeted onto and ready for review . 
 What value does sorting these add ? 
 After discussion with cjpatton , I tacked on a commit that attempts to specify this stuff without defining and inserting it into various messages . Instead , we now require a consistent ordering between and ( the order of does n't matter ; the collector does n't need to do anything special with the leader 's share ) . I 'm not sure if this is better than the approach but now we can compare them . 
 Taking a step back , is it true that we 're simply trying to address aggregator routing at the leader here ? In particular , upon receipt of a Report , the leader needs to know what aggregators should get the shares contained therein . The latest commit does this by saying " send the i - th share to the i - th helper that was agreed upon in the Param structure . " Is this a correct interpretation of the problem ? 
 This PR is about addressing : putting enough structure into the PDUs so that we could extend the protocol in the future to support multiple helpers without changing the wire format of messages , even if actually supporting 1 helper might require more rounds of communication and new , yet undefined messages . IMO , that also requires solving , the problem of matching an encrypted input share to an aggregator . I also think that now that we have introduced a means of distinguishing arbitrary numbers of aggregators from each other , we should use it to diversify key material ( but see my question below ) . The actual protocol extensions are , which this PR does not address , nor do I think we intend to do anything about that issue anytime soon . Yes . I apologize for the back and forth on this . I think we have two important questions to resolve in this PR : We currently diversify keys by " server role " , which is either " leader " or " helper " . Now that we admit that there can be more than two server roles , do we want to further diversify key material by that role ? If yes to 1 . , do we want to do that in this PR or should we punt to a subsequent change and leave for leader , for helper ? We want some protocol messages to be able to contain a vector of sub - messages , one for each participating aggregator . Do we want to achieve that with a vector of labeled tuples ( as in commit ) , or with a simple vector of values , where the index into the vector is effectively the aggregator ID ( as in commit ) ? I think both are workable , with different tradeoffs . I prefer the labeled tuples approach because I think it is more explicit and imposing fewer requirements on the order of lists in different messages means less validation code and removes opportunities for implementation bugs . argues for the simple vector approach . I 'm not sure how to resolve that disagreement , except that consensus matters more to me than which one we choose . Should we , uh , vote on it or something ? 
 No worries -- we 're in alignment on the problem here . As for the questions : 1 ) Further diversification seems not useful , and appears like it only complicates matters . In particular , a helper does n't necessarily need to know that it 's helper i or j -- it just needs to know that it 's not the leader . Sticking with what we currently have makes sense . 2 ) I claim this is about routing messages to aggregators . As a silly example , imagine we split up Report into multiple HTTP requests , where each request carried one share for each aggregator . The request could identify the aggregator . This would also solve the problem here , but perhaps in a ( much ) worse way . Both proposals -- tagging each share by an ID , or sorting the report based on the parameters -- seem fine , but I lean towards something like the former . Sorting the report means that everyone needs to agree on the order of helpers everywhere , which seems like an unnecessary constraint . I would prefer a design wherein each share unambiguously , and without additional coordination , identifies the intended aggregator . 
 SGTM 
 In the interest of continuing to move forward , I am going to keep the version where we keep the ordering consistent across different messages . Per observation , I 've punted the question of aggregator IDs or server roles beyond the existing 0x01/0x00 constants to . 
 Currently , the derivation of HPKE context entangles a constant for , which is 0x01 for the leader and 0x00 for the helper . Some iterations of PR , which aimed to generalize message definitions to support more than one helper , introduced a notion of an aggregator ID or role , and then used that role in the key derivation . That notion wound up being a little more complicated than what needed to be addressed in that PR , but we should still consider whether and how we want to formalize the notion of an aggregator 's role and whether it should be involved in encryption key derivation . 
 I 'm not sure if this should go in label parking - lot . The server role constants we have now are sufficient for the deployments that will realistically materialize in the near or medium term , and I think we could define server roles in the future in a manner that is compatible with the existing constants . 
 I 'm in favor of the generalization , but am fine with putting this in the parking lot . 
 One important question we must address is the maximum number of aggregators in a protocol instantiation . We are already encountering places where . 
 We 've long since introduced a formal enumeration whose layout was chosen to allow adding more helpers in the future . Closing . 
 We have this definition : The intent is to eventually allow an arbitrary number of aggregators , but how is the leader meant to know which goes to which aggregator ? Is share 0 understood to be the leader 's , and share 1 is the helper 's ? 
 Yeah , that 's 0 and 1 . But 3 and 4 ? LOL . Maybe we should do : Where is TBD . 
 Ooh , vectors of labeled tuples are much better than my idea of requiring consistent ordering of per - aggregator values across , and 
 This is ( going to be ) resolved by . 
 This seems to be a dupe fo -- can we close ? 
 Per discussion in the call today , I think this issue is about updating the various PDU definitions to allow for multiple helpers . For instance , currently has fields and , but it should instead have . is the more complex problem of extending the protocol to handle 1 helper , which probably involves more rounds of communication . I 'm going to grab this issue because I ran into which I think is related , so I 'll fix both . 
 The current specification describes a PPMParam struct and defines a TaskID as SHA-256(PPMParam ) . However , because the PPMParam is never rendered on the wire . Rather it is just serialized in order to be fed into SHA-256 . The parameters are provided to the endpoints out of band . Consider a client implementor who behaves as follows : Create a JSON structure J that defines the parameters . Translate J into    the client the pair [ J , TaskID = SHA-256(P ) ] You 'll note that the client never sees the PPAParam at all . As far as I can tell , this would be a completely conformant implementation . Similar comments apply to the aggregators as we also do not define how they are configured . So , similarly they could just be sent  and have a TaskID -JSON lookup table . This is n't just a possible implementation , but it 's likely to be a common one , because people have whatever central configuration system they have and do n't necessarily want to carry around some binary blob . In fact , the only reason people need to see the PPAParam structure at all as opposed to the information in it is to hash it to create TaskID ( yes , i appreciate that I suggested the hashing thing ) . So what I suggest is we merely make a list of the things that the client and the aggregators need to know respectively and not define any structure that contains that list . Then we can just pick the TaskID randomly . When we originally discussed hashing , the idea was to have parameters be transitively incorporated into the protocol to avoid situations where each side is using different parameters . It 's not clear to me that this is that important . A malicious client can just lie and use its own parameters , and if there is a bug , then it 's just as likely it involves misinterpreting the parameters . 
 I think our goal was to force clients to commit to a set of PPMParams , but you 're right , nothing really ties the actual behavior of clients or aggregators to the PPMParams . We put a 16 byte nonce into PPMParam with the intention of guaranteeing that  would be unique , but I think we can just instead have  be that nonce . As for the sizes , we picked len(PPMTaskID ) = 32 to match the length of a SHA-256 , and len ( ) = = 16 to match a v4 UUID . If  is going to be randomly generated instead of derived , then 16 bytes should suffice . 
 The problem we 're trying to solve by binding the PPM params to encryption has less to do with malicious clients than honest ones . Namely , it 's meant to ensure that honest aggregators reject reports that were generated with incorrect parameters . As to whether this binding is strictly necessary , I agree that it 's not clear that there 's an attack it prevents right now . This binding is conservative in the sense that it aims to mitigate attacks that may come up in the future , as the set of parameters evolve . Another benefit of this binding is that it ensures the client knows what privacy - sensitive parameters are enforced by honest aggreagators . Namely , the batch size , minimum batch window , and , eventually , parameters for DP . To conclude , there may not be a need to define a serialization format for PPM params , and it would be fine to just randomly the task ID . However , there should be some sort of binding between the params and HPKE encryption . 
 You seem to be assuming a very specific implementation , but nothing in the spec requires that , as shown by my example above . 
 Based on our conversation in the call on 8/18 , I think the consensus is we should not define but instead informally list the parameters that all participants must agree on ( which does appear in several on - the - wire messages ) will become a randomly generated sequence of bytes instead of being a SHA-256 of the Unless someone objects ( tagging to make sure he sees this issue as he has n't participated yet ) I 'm going to make this doc change this week , after landing . 
 seems like it complicates this , since we 'd have to spell out some way to sort aggregators . Do we want to land this change before that one ? 
 I want to land first , since it 's just about done . I do n't think it 'll be too hard to change references to into references to an ordered list of aggregators . 
 Sounds good ! 
 I 'm also a bit concerned about universal sorting , but I think we can resolve this separately , b / c it 's really just about what goes into the HPKE functions . I suggest we land this once the more minor comments have been resolved .
Paragraph (version A):
"leader_url": The leader's endpoint URL.
Paragraph (version A):
"helper_url": The helper's endpoint URL.
Paragraph (version B):
"aggregator_endpoints": Vector of URLs relative to which an
aggregator's API endpoints can be found.  The leader's endpoint
MUST be the first in the vector.  The order of the
"encrypted_input_shares" in a "Report" (see uploading-reports)
MUST be the same as the order in which aggregators appear in this
vector.
Paragraph (version A):
"encrypted_input_shares" contains the encrypted input shares of
each of the aggregators.
Paragraph (version B):
"encrypted_input_shares" contains the encrypted input shares of
each of the aggregators.  The order in which the encrypted input
shares appear MUST match the order of the "aggregator_endpoints"
in the "Param" corresponding to "task_id" (i.e., the first share
should be the leader's, the second share should be for the first
helper, and so on).
Paragraph (version A):
where "input_share" is the aggregator's input share and "task_id",
"time", and "nonce" are the fields of "Report".
Paragraph (version B):
where "input_share" is the aggregator's input share and "time",
"nonce" and extensions are the corresponding fields of "Report".
Paragraph (version A):
Once a set of clients have uploaded their reports to the leader, the
leader can send them to the aggregators to be verified and
aggregated.  In order to enable the system to handle very large
batches of reports, this process can be performed incrementally.  To
aggregate a set of reports, the leader sends an AggregateReq to the
helper containing those report shares.  The helper then processes
them (verifying the proofs and incorporating their values into the
ongoing aggregate) and replies to the leader.
Paragraph (version B):
Once a set of clients have uploaded their reports to the leader, the
leader can send them to the helpers to be verified and aggregated.
In order to enable the system to handle very large batches of
reports, this process can be performed incrementally.  To aggregate a
set of reports, the leader sends an AggregateReq to each helper
containing those report shares.  The helper then processes them
(verifying the proofs and incorporating their values into the ongoing
aggregate) and replies to the leader.
Paragraph (version A):
The "time", "nonce", and "extensions" fields have the same value as
those in the report uploaded by the client.  Similarly, the
"helper_share" field is the helper's encrypted input share as it
appeared in the report.  [OPEN ISSUE: We usually only need to send
this in the first aggregate request.  Shall we exclude it in
subsequent requests somehow?]  The remainder of the structure is
dedicated to the protocol-specific request parameters.
Paragraph (version B):
The "time", "nonce", and "extensions" fields have the same value as
those in the report uploaded by the client.  Similarly, the
"helper_share" field is the "EncryptedInputShare" from the "Report"
whose index in "Report.encrypted_input_shares" is equal to the index
of "[aggregator]" in "Param.aggregator_endpoints".  [OPEN ISSUE: We
usually only need to send this in the first aggregate request.  Shall
we exclude it in subsequent requests somehow?]  The remainder of the
structure is dedicated to the protocol-specific request parameters.
Paragraph (version A):
where "sk" is the HPKE secret key and "server_role" is the role of
the server ("0x01" for the leader and "0x00" for the helper).  If
decryption fails, then the sub-response consists of a "decryption
error" alert.  [See issue#57.]  Otherwise, the helper handles the
request for its plaintext input share "input_share" and updates its
state as specified by the PPM protocol.
Paragraph (version B):
where "sk" is the HPKE secret key, "task_id" is
"AggregateReq.task_id" and "server_role" is the role of the server
("0x01" for the leader and "0x00" for the helper). "time", "nonce"
and "extensions" are obtained from the corresponding fields in
"AggregateSubReq".  If decryption fails, then the sub-response
consists of a "decryption error" alert.  [See issue#57.]  Otherwise,
the helper handles the request for its plaintext input share
"input_share" and updates its state as specified by the PPM protocol.
Paragraph (version A):
to the helper.  The helper responds to this request by extracting its
output share from its state and encrypting it under the collector's
HPKE public key.
Paragraph (version B):
to each helper.  The helper responds to this request by extracting
its output share from its state and encrypting it under the
collector's HPKE public key.
Paragraph (version B):
where "pk" is the HPKE public key encoded by the collector's HPKE key
configuration, "task_id" is "OutputShareReq.task_id" and
"aggregator_id" is the index of the helper's endpoint in
"Param.aggregator_endpoints". "output_share" is the serialized
"OutputShare". "batch_start" and "batch_end" are obtained from the
"OutputShareReq".
Paragraph (version A):
"leader_share" and "helper_share" are "EncryptedOutputShare"s as
described in output-share-request, except that the "task_id",
"batch_start", and "batch_end" used to encrypt the "OutputShare"
are obtained from the "CollectReq".
Paragraph (version B):
"shares" is a vector of "EncryptedOutputShare"s, as described in
output-share-request, except that for the leader's share, the
"task_id", "batch_start", and "batch_end" used to encrypt the
"OutputShare" are obtained from the "CollectReq".