need some error handling to catch cases where:
  download fails (retry?)
  download fails + all retries fail:  what do we do? -> cleanup
  server dies while downloading (need persistent state of whether shard was consistent... file marker?)