Draft: draft-gpew-priv-ppm

Commentary: Make the handling of collect requests on the leader asynchronous , since servicing a collect request could entail running the preparation protocol over all the reports that fall into the batch interval , especially in VDAFs like poplar1 where the aggregation parameter is not available until the collector provides it . I 've only made the collect request handling asynchronous . My reasoning is that handling other requests synchronously should be feasible : upload requests should only contain shares of a single report and thus never be all that big . Additionally , the client needs an explicit acknowledgement from the aggregator that the uploaded report was accepted and has been persisted . the requests in the aggregation / preparation sub - protocol should also never get that big : even when preparing a large number of reports , the leader can choose to carve that work up into any number of aggregation jobs , each of which covers a small enough number of reports that the message sizes and processing times should be reasonable to handle in synchronous request / response exchanges . 
 With this change , we will have two differing definitions of structs named . My understanding is that clients will know which one to expect based on the URL of the request , and the status code in the response . ( they will receive a URL from /collect on success , and the URL will return a URL with status code 202 , or shares with status code 200 ) I think we should change one of these for clarity to the reader , i.e. or . 
 It 's easy to imagine that a request will take a long time , especially if you are using Collect and that has to trigger a bunch of work from the leader . We need some way of handling this . 
 This intersects with something we were discussing last week , which is whether or not should have in it , or a URL from which collector config can be fetched dynamically . We decided on the former to spare collectors of running an online HTTP server . However , if we adopt an asynchronous model for collect requests , then we might need collectors to be an HTTP server anyway so that the leader can notify them that a collect request is complete , in which case we should make collectors have a endpoint like the other servers . 
 and I were talking about this today . We agree that / definitely needs to be made asynchronous since in the case , servicing a will definitely incur lots of expensive work between the aggregators ( this is less likely in the case because the aggregators are n't blocked on obtaining the aggregation parameter from the collector in that VDAF ) . What I 'm not sure if is whether we should make all the interactions between the aggregators during preparation and verification asynchronous , and I 'm not sure how to decide . It seems like it should be possible for leaders to scope each aggregate requests to a small enough number of shares that each one can reasonably be serviced synchronously . 
 For collect requests , I 'd also note here that asynchronicity is probably going to be important for , as it 's possible that the aggregation flow will fall behind the upload flow . ( Imagine the helper can only handle 1000 reports / sec , but the clients are uploading 1million reports / sec . ) As for aggregate requests , I would expect these to be able to be completed fairly quickly ( in a matter of seconds , in the worst case ) . Based on our experience so far , the most expensive part appears to be interacting with long - term storage , which is required for anti - replay and , if we take , looking up report shares . I think we can hold off on making these requests asynchronous until we 've gotten enough deployment experience to determine if this is going to be a problem . 
 provided some interesting prior art that we should rely on when addressing this : 
 Nicely done .
Mailing list commentary: Hi , Maybe this might help you : ( URIs , not UUIDs ) You might also find this informative : You are probably familiar with how ACME does asynchronous processing , which is largely OK . This probably comes down to preference at some level . However , asynchronous interactions are a fair bit more complex than synchronous ones , so avoiding that complexity is good . Keep in mind that modern HTTP versions are n't stalled if there are a handful of long - running requests , so unless you are talking a fairly significant time , it might be cleaner and easier to just have the client wait on hold . Asynchronous requests do come with some benefits to go with the complexity . For instance , robustness can be improved if the initial request creates resources that can be referred to later on a different connection without having to rely on the connection remaining viable . RFC 7240 does suggest that you might offer both , but I would only go that far if you thought that implementations would consistently use both options .
Paragraph (version A):
Depending on the VDAF scheme and how the leader is configured, the
collect request may cause the leader to send a series of aggregate
requests to the helpers in order to compute their share of the
aggregate.  Alternately, the leader may already have made these
requests and can respond immediately.  In either case it responds to
the collector's request as follows.
Paragraph (version B):
Depending on the VDAF scheme and how the leader is configured, the
leader and helper may already have prepared all the reports falling
within "batch_interval" and be ready to return the aggregate shares
right away, but this cannot be guaranteed.  In fact, for some VDAFs,
it is not be possible to begin preparing inputs until the collector
provides the aggregation parameter in the "CollectReq".  For these
reasons, collect requests are handled asynchronously.
Paragraph (version B):
Upon receipt of a "CollectReq", the leader begins by checking that
the request meets the requirements of the batch parameters using the
procedure in batch-parameter-validation.  If so, it immediately sends
the collector a response with HTTP status 303 See Other and a
Location header containing a URI identifying the collect job that can
be polled by the collector, called the "collect job URI".
Paragraph (version B):
The leader then begins working with the helper to prepare the shares
falling into "CollectReq.batch_interval" (or continues this process,
depending on the VDAF) as described in pa-aggregate.
Paragraph (version B):
After receiving the response to its CollectReq, the collector makes
an HTTP GET request to the collect job URI to check on the status of
the collect job and eventually obtain the result.  If the collect job
is not finished yet, the leader responds with HTTP status 202
Accepted.  The response MAY include a Retry-After header field to
suggest a pulling interval to the collector.
Paragraph (version B):
Once all the necessary reports have been prepared, the leader obtains
the helper's encrypted aggregate share for the batch interval by
sending an "AggregateShareReq" to the helper as described in
aggregate-share-request.  The leader then computes its own aggregate
share by aggregating all of the prepared output shares that fall
within the batch interval.
Paragraph (version B):
If obtaining aggregate shares fails, then the leader responds to
subsequent HTTP GET requests to the collect job URI with an HTTP
error status and a problem document as described in errors.
Paragraph (version B):
The leader MUST retain a collect job's results until the collector
sends an HTTP DELETE request to the collect job URI, in which case
the leader responds with HTTP status 204 No Content.