Repository: pantsbuild/pants

Issue:
Consider hosting pantsbuild binaries ourselves.
We've had outages twice due to bintray rate limits as well as smaller spot outages due to bintray general outages. It may behoove us to serve our binaries, say via Amazon S3, if the cost is low enough.


Discussion:
I'm going to poke at this to test difficulty / maintainability / cost. It sure would be nice to just rely on bintray instead.
OK, added a `binaries` CNAME to our pantsbuild.org DNS that points to `binaries.pantsbuild.org.s3.amazonaws.com` and this allows for serving from a new `binaries.pantsbuild.org` bucket in S3. I manually uploaded https://github.com/pantsbuild/binaries/README.md to the bucket with public read as the proof of concept: http://binaries.pantsbuild.org/README.md. Next up, scripting a sync routine from a https://github.com/pantsbuild/binaries clone to the bucket. 
OK, created a free-tier EC2 host using the amazon-linux image and a `binaries.baseurls` of https://s3.amazonaws.com/binaries.pantsbuild.org does the trick nicely:
```
$ ssh -i ~/.ssh/xxx yyy@zzz.compute-1.amazonaws.com
$ yum install git
$ git clone https://github.com/pantsbuild/binaries
$ cd binaries/build-support
$ aws configure
AWS Access Key ID [None]: xxx
AWS Secret Access Key [None]: xxx
Default region name [None]: 
Default output format [None]: 
$ aws s3 sync --acl public-read . s3://binaries.pantsbuild.org
```
I'll prep a PR with a switch in either the default `binaries.baseurls` or else in the pantsbuild/pants repo's configured one after ammending the pantsbuild/binaries README to include the procedure to sync there - or better, after adding a `sync-s3.sh` script.
Added the `node-preinstalled-modules.pantsbuild.org` S3 bucket and synced `https://github.com/pantsbuild/node-preinstalled-modules` as per above to work around similar issues with the node tests in CI.

Via:
```
$ git clone https://github.com/pantsbuild/node-preinstalled-modules
$ cd node-preinstalled-modules
$ aws s3 sync --exclude ".git/*" --acl public-read . s3://node-preinstalled-modules.pantsbuild.org
```
Instructions for work-around sent to pants-devel@ here: https://groups.google.com/d/topic/pants-devel/q4MYb7QrWA4/discussion
Added `sync-s3.sh` scripts to [binaries](https://github.com/pantsbuild/binaries/commit/ee401a6fa97870362e92533aca907d2a62b77ada) and [node-preinstalled-modules](https://github.com/pantsbuild/node-preinstalled-modules/commit/3f7228a07697220f5fab5f606a56d71b8c3cf61c).
The `pantsbuild` AWS IAM user Access Key ID and AWS Secret Access Key ID have been added to the 1Password vault.
Reopening to track work on native engine publishing.
Is the account with the S3 files also on the free tier? If so, I'd be a bit nervous using it by default - number of allowed requests is limited.
@viveksjain it is not, we're paying for the s3 bucket
Great, thanks
`Exception message: Failed to fetch binary bin/native-engine/mac/10.12/af2e5b09cb7eee743c8d4e065258ac046c8a19ac/native_engine.so from any source: (Failed to fetch binary from https://dl.bintray.com/pantsbuild/bin/build-support/bin/native-engine/mac/10.12/af2e5b09cb7eee743c8d4e065258ac046c8a19ac/native_engine.so: Fetch of https://dl.bintray.com/pantsbuild/bin/build-support/bin/native-engine/mac/10.12/af2e5b09cb7eee743c8d4e065258ac046c8a19ac/native_engine.so failed with status code 403)`

pants version is 1.3.0.
Thanks @tglstory 
I'll be handling the OSX upload of the native-engine binary. No ETA yet... stay tuned.
OK - linux binaries for the native_engine.so are uploaded for these releases:
```
release_1.3.0
release_1.3.0.dev10
release_1.3.0.dev11
release_1.3.0.dev12
release_1.3.0.dev13
release_1.3.0.dev14
release_1.3.0.dev15
release_1.3.0.dev15-republish
release_1.3.0.dev16
release_1.3.0.dev16-republish
release_1.3.0.dev17
release_1.3.0.dev18
release_1.3.0.dev19
release_1.3.0.dev3
release_1.3.0.dev4
release_1.3.0.dev5
release_1.3.0.dev6
release_1.3.0.dev8
release_1.3.0.dev9
release_1.3.0dev0
release_1.3.0dev1
release_1.3.0dev2
release_1.3.0rc0
release_1.3.0rc1
release_1.3.0rc2
release_1.3.0rc3
release_1.3.0rc4
release_1.4.0.dev0
release_1.4.0.dev1
release_1.4.0.dev2
release_1.4.0.dev3
release_1.4.0.dev4
release_1.4.0.dev5
release_1.4.0.dev6
release_1.4.0.dev7
```
OSX binaries are currently syncing: the following have been pushed:
```
release_1.3.0
release_1.3.0rc4
release_1.3.0rc3
release_1.3.0rc2
release_1.3.0rc1
release_1.3.0rc0
release_1.4.0.dev3
release_1.4.0.dev2
release_1.4.0.dev1
release_1.4.0.dev0
release_1.3.0.dev19
release_1.3.0.dev18
release_1.3.0.dev17
release_1.3.0.dev16
release_1.3.0.dev15
```

EDIT: The remaining versions have now been pushed:
```
release_1.3.0.dev14
release_1.3.0.dev13
release_1.3.0.dev12
release_1.3.0.dev11
release_1.3.0.dev10
release_1.3.0dev2
release_1.3.0dev1
release_1.3.0dev0
release_1.3.0.dev9
release_1.3.0.dev8
release_1.3.0.dev6
release_1.3.0.dev5
release_1.3.0.dev4
release_1.3.0.dev3
```
Alright... if you've applied the `pants.ini` configuration that @jsirois demonstrated here: https://github.com/pantsbuild/pants/pull/4801/commits/8daaaae67093bef00c5f716af05b20ecdd43427d , you should no longer be experiencing this issue. If you do, please report on a _new_ ticket... this one will be used for tracking the long term fix.

Resolving patch:
diff --git a/src/python/pants/binaries/binary_util.py b/src/python/pants/binaries/binary_util.py
--- a/src/python/pants/binaries/binary_util.py
+++ b/src/python/pants/binaries/binary_util.py
@@ -55,7 +55,7 @@ class Factory(Subsystem):
     @classmethod
     def register_options(cls, register):
       register('--baseurls', type=list, advanced=True,
-               default=['https://dl.bintray.com/pantsbuild/bin/build-support'],
+               default=['https://s3.amazonaws.com/binaries.pantsbuild.org'],
                help='List of urls from which binary tools are downloaded.  Urls are searched in '
                     'order until the requested path is found.')
       register('--fetch-timeout-secs', type=int, default=30, advanced=True,


Test patch: