Initial commit of datamanager.py command line utility. So far all information comes from the python SDK. Future versions will talk to the keep servers as well.
Updated message to be clearer.
Deleted unused code, added code to read blocks on keep server.
Merge branch 'master' of git.curoverse.com:arvados into 1968-monitor-disk-usage
Added manifest size to calculations.
Modified algorithm to take disk replication when counting usage.
Added logging to datamanager, as a step towards writing http server.
Moved main flow into loadAllData method to get ready for multithreading.
Started using global to actually assign to global variables instead of local variables that shadow them. Added webserver and threads!
Added user page. Started using send_error. Added port flag.
Added handler for collections. Also renamed byte_size to byteSize to follow method naming convention.
Added ability to disable server by specifying port 0. Started keeping api reponse just in case. Started tracking replication level for peristers (although still working on using it everywhere). Added maxdict to make replication tracking easier.
Started reporting unweighted persist usage based on requested replication, not actual replication. refs #2572
Added method to computed weighted cost of different replication levels. Added main method to aid in unit testing.
Added unittests for computeWeightedReplicationCosts. Moved arg parsing to main block so that I can use commandline flags for tests when testing. Fixed lots of bugs surfaced by tests.
Added memoization to computeWeightedReplicationCosts.
Added weighted persist disk usage. Removed debugging line I had forgotten. refs #2572.
Added disk usage logging! logs are written to api server.

Switched logic for readers, actual disk space is no longer examined,
instad it is the sum of the byte size of the collections (including
the manifest size) they requested to read. In rertrospect I should
probably report both the requested size and the size of the blocks
that actually appear on disk.

split reportUserDiskUsage() into computeUserStorageUsage() and
printUserStorageUsage(). Added logUserStorageUsage() to do the logging
mentioned above.

Moved command line argument parser out of the __main__ block so that
it can be accessed by other packages who wish to support commandline
arguments for this package even though they may have their own main
block.

Added new command line args for api server logging.

refs #2579 #1757
Added reporting of persisted blocks which are under or over replicated. Also added reporting of blocks without collections. refs #2574
Removed hack which was compensating for keep server bug which has been fixed.
Update to match new log format info field has been renamed to properties.