Add AG disk curation tool
Improved shardcounter API--add asynchronous increment, decrement, and
delete operations, and do without the shard config class (since this
role is fulfilled by the Volume)
Make default batch size 6, and default page size 512
MS: Do not require the caller to pass a cursor to get back a particular
page's entries.
MS: have directories use shard counters to track how many children they
have, so when we do a ListDir(), we can always tell when the caller need
to try again due to the datastore not having consistent query results.
MS: map page IDs to cached cursors internally. Do not require clients to
pass back cursors, since they could break our caching implementation.
Pass flow control data hints from the MS to the client.
Do not pass cursors or "have_more" to the client.  Always have the
client request more page data (by page ID), until there are no entries
returned (indicating EOF)
Add large disk dataset for testing the AG
Don't track how many files there are in a volume.  It doesn't really
matter to us
* remove num_files from the volume protocol buffer
* add flow control to registration protocol buffer
AG tools: Get number of threads and retry counts using the AG python
library.
Add method to allow a curation tool to extract the number of threads and
number of retries from a curation tool's default arguments.
Pass the correct argument when generating a specfile's Pair data
Fill in AG disk curation python library
Don't expect disk query strings to contain the absolute paths to the datasets.
Use the config-given dataset root instead.
Updated index with the queries we'll run
Give a maximum specfile size
Remove page cursors from the MS FILE URL
Ensure that a download context is not referenced by a download set when
freeing it.  Also, set pending and cancelling flags only when the
downloader is locked.  Additionally, perform NULL checks explicitly,
explicitly close braces.
Don't guess an output size when inflating.  Have the caller pass a
maximum decompressed size instead.