Use CacheControl instead of custom cache code

* [ ] Bundle CacheControl and lockfile
* [ ] Use cross platform standard path locations
* [ ] Some way to disable caching?
* [ ] Soft fail if we cannot write to our cache directory
* [x] Deprecate the --download-cache option & removes the download cache code.
* [x] Remove the in memory page cache on the index
* [x] Use CacheControl to cache all cacheable HTTP requests to the filesystem.
  * Properly handles CacheControl headers for unconditional caching.
  * Use ETag and Last-Modified headers to attempt to do a conditional HTTP request to speed up cache misses and turn them into cache hits.
  * Remove some concurrency unsafe code in the download cache accesses.

Closes #1732
Closes #1734
Closes #1634
Closes #1673
Closes #1141
Closes #1287
Closes #685