Repository: DataBiosphere/toil

Issue:
toil AWS Provisioner failing without TOIL_AWS_AMI set
Hi - 

Just getting started with toil (3.5.1), and attempted to set up a cluster as per the "Autoscaling" documentation:

`toil launch-cluster -p aws <cluster-name> \
     --nodeType=m3.large \
     --keyPairName=<keypair>`

Consistently fails while trying to launch a non-preemptable leader as follows:
`Traceback (most recent call last):
  File "/usr/local/bin/toil", line 9, in <module>
    load_entry_point('toil==3.5.1', 'console_scripts', 'toil')()
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilLaunchCluster.py", line 53, in main
    keyName=config.keyPairName, spotBid=spotBid)
  File "/usr/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 357, in launchCluster
    create_ondemand_instances(ctx.ec2, image_id=cls._discoverAMI(ctx),
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bd2k/util/__init__.py", line 45, in new_f
    r = f( *args )
  File "/usr/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 120, in _discoverAMI
    assert len(coreOSAMI) == 1
AssertionError
`

Searching through the AWS console, there doesn't appear to be to be an AMI matching the criteria this function defaults to - description containing 'stable 1068.9.0' with an owner '679593333241'. 

Updating the description string to 'stable 1235.6.0' will allow cluster creation to proceed after going to a URL to accept licensing terms.
toil AWS Provisioner failing without TOIL_AWS_AMI set
Hi - 

Just getting started with toil (3.5.1), and attempted to set up a cluster as per the "Autoscaling" documentation:

`toil launch-cluster -p aws <cluster-name> \
     --nodeType=m3.large \
     --keyPairName=<keypair>`

Consistently fails while trying to launch a non-preemptable leader as follows:
`Traceback (most recent call last):
  File "/usr/local/bin/toil", line 9, in <module>
    load_entry_point('toil==3.5.1', 'console_scripts', 'toil')()
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilLaunchCluster.py", line 53, in main
    keyName=config.keyPairName, spotBid=spotBid)
  File "/usr/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 357, in launchCluster
    create_ondemand_instances(ctx.ec2, image_id=cls._discoverAMI(ctx),
  File "/usr/local/Cellar/python/2.7.12_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/bd2k/util/__init__.py", line 45, in new_f
    r = f( *args )
  File "/usr/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 120, in _discoverAMI
    assert len(coreOSAMI) == 1
AssertionError
`

Searching through the AWS console, there doesn't appear to be to be an AMI matching the criteria this function defaults to - description containing 'stable 1068.9.0' with an owner '679593333241'. 

Updating the description string to 'stable 1235.6.0' will allow cluster creation to proceed after going to a URL to accept licensing terms.


Discussion:
@rborkow what EC2 zone are you running in? Unfortunately, AMIs are local to every zone so it is possible that the zone you are running in doesn't have the CoreOS AMI we rely on. 
Got it -- 'us-west-1'
Ahh, thats unfortunate. I thought I had checked that region for a matching AMI - I hope they aren't deprecating it. Let me know if 1235.6.0 is working for you - it might make sense for us update to that version. 
Just checked us-west-2 and us-east-1 and they're all failing in the same manner.

Using 1235.6.0, spent ~20 minutes on retrying "toil.provisioners.aws.awsProvisioner: Waiting for docker on <host-ip> to start..." before giving up. SSH'd into the host, no process named "docker daemon", but "dockerd" was present. Updating that string in `_waitForDockerDaemon()` allowed the provisioner to complete, and `toil ssh-cluster <cluster-name>` succeeds.
@rborkow I can still see the AMI in us-west-2, our preferred region. Maybe this is an account-related thing rather than region-related..
@cket Here's my search output in us-west-2:
<img width="1470" alt="screen shot 2017-01-31 at 1 04 25 pm" src="https://cloud.githubusercontent.com/assets/2956000/22484397/1e1e259a-e7b6-11e6-9187-879aac6f9af9.png">

@rborkow thats very strange.. I just spun up a test EC2 instance in us-west-2 so I'm sure it's still available for us
I'm also running into this issue launching a Toil cluster in us-east-1, and similarly don't see the older CoreOS AMIs. If I manually search, I see CoreOS-stable-1185.5.0 and CoreOS-stable-1235.4.0 to CoreOS-stable-1235.9.0. Trying to use the most recent stable AMI results in:
```
$ export TOIL_AWS_AMI=ami-61659e77
$ toil launch-cluster -p aws bcbio --nodeType=t2.small --keyPairName=bcbio --zone us-east-1d
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.utils.toilLaunchCluster: Using aws provisioner.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker name of toil as TOIL_DOCKER_NAME is not set.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker appliance of quay.io/ucsc_cgl/toil:3.7.0a1-6a452b988d092f55f6da966de8248dd9c1a53a85 as TOIL_APPLIANCE_SELF is not set.
kunkel 2017-02-12 09:49:30,032 MainThread INFO toil.provisioners.aws.awsProvisioner: Launching non-preemptable leader
kunkel 2017-02-12 09:49:30,032 MainThread INFO cgcloud.lib.ec2: Creating t2.small instance(s) ...
Traceback (most recent call last):
  File "/home/chapmanb/install/bcbio-vm/anaconda/bin/toil", line 11, in <module>
    load_entry_point('toil==3.7.0a1', 'console_scripts', 'toil')()
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/utils/toilLaunchCluster.py", line 73, in main
    keyName=config.keyPairName, spotBid=spotBid, userTags=tagsDict, zone=config.zone)
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/provisioners/aws/awsProvisioner.py", line 361, in launchCluster
    spec=kwargs, num_instances=1)
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/cgcloud_lib-1.6.0-py2.7.egg/cgcloud/lib/ec2.py", line 416, in create_ondemand_instances
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>VPCResourceNotSpecified</Code><Message>The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.</Message></Error></Errors><RequestID>513f943f-a03d-44c4-8790-4acd4dc6d29f</RequestID></Response>
```
I'm not sure if this is an AMI choice issue or something incorrect in my setup that we're not launching these inside a VPC. I didn't see an option for specifying a VPC. Thanks for any tips/tricks.
Following up on my own post, I dug into the code and realized it assumes a default VPC. I have an older AWS account without default VPCs, so I will submit a PR to allow users to supply a VPC subnet ID to create instances in.

With that in place, I can get further and start towards launching (still stuck on waiting for Docker but will debug that separately).

On topic for this thread, why is CoreOS chosen here? The manual click through licensing is an extra annoying step if you like automation, and the region availability seems problematic. Could we accomplish what you need for Toil with a standard Ubuntu image which is replicated across all the regions and has consistent and long term availability? I've had good luck with these: http://cloud-images.ubuntu.com/locator/ec2/
@chapmanb CoreOS has Docker pre-installed, which we need for the provisioner since it is based on the Toil/Mesos Docker container. The licensing is once per account so it shouldn't bother you again. 

I'm looking into the AMI problem, it is still showing up on the UCSC account but I can't see it on my private account.
I'm not sure what is going on here - I found a matching AMI but it has the wrong 'Owner' attribute. I don't know why a public AMI's visibility would be affected by different accounts. 
@rborkow what EC2 zone are you running in? Unfortunately, AMIs are local to every zone so it is possible that the zone you are running in doesn't have the CoreOS AMI we rely on. 
Got it -- 'us-west-1'
Ahh, thats unfortunate. I thought I had checked that region for a matching AMI - I hope they aren't deprecating it. Let me know if 1235.6.0 is working for you - it might make sense for us update to that version. 
Just checked us-west-2 and us-east-1 and they're all failing in the same manner.

Using 1235.6.0, spent ~20 minutes on retrying "toil.provisioners.aws.awsProvisioner: Waiting for docker on <host-ip> to start..." before giving up. SSH'd into the host, no process named "docker daemon", but "dockerd" was present. Updating that string in `_waitForDockerDaemon()` allowed the provisioner to complete, and `toil ssh-cluster <cluster-name>` succeeds.
@rborkow I can still see the AMI in us-west-2, our preferred region. Maybe this is an account-related thing rather than region-related..
@cket Here's my search output in us-west-2:
<img width="1470" alt="screen shot 2017-01-31 at 1 04 25 pm" src="https://cloud.githubusercontent.com/assets/2956000/22484397/1e1e259a-e7b6-11e6-9187-879aac6f9af9.png">

@rborkow thats very strange.. I just spun up a test EC2 instance in us-west-2 so I'm sure it's still available for us
I'm also running into this issue launching a Toil cluster in us-east-1, and similarly don't see the older CoreOS AMIs. If I manually search, I see CoreOS-stable-1185.5.0 and CoreOS-stable-1235.4.0 to CoreOS-stable-1235.9.0. Trying to use the most recent stable AMI results in:
```
$ export TOIL_AWS_AMI=ami-61659e77
$ toil launch-cluster -p aws bcbio --nodeType=t2.small --keyPairName=bcbio --zone us-east-1d
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.lib.bioio: Root logger is at level 'INFO', 'toil' logger at level 'INFO'.
kunkel 2017-02-12 09:49:29,140 MainThread INFO toil.utils.toilLaunchCluster: Using aws provisioner.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker name of toil as TOIL_DOCKER_NAME is not set.
kunkel 2017-02-12 09:49:30,031 MainThread INFO toil: Using default docker appliance of quay.io/ucsc_cgl/toil:3.7.0a1-6a452b988d092f55f6da966de8248dd9c1a53a85 as TOIL_APPLIANCE_SELF is not set.
kunkel 2017-02-12 09:49:30,032 MainThread INFO toil.provisioners.aws.awsProvisioner: Launching non-preemptable leader
kunkel 2017-02-12 09:49:30,032 MainThread INFO cgcloud.lib.ec2: Creating t2.small instance(s) ...
Traceback (most recent call last):
  File "/home/chapmanb/install/bcbio-vm/anaconda/bin/toil", line 11, in <module>
    load_entry_point('toil==3.7.0a1', 'console_scripts', 'toil')()
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/utils/toilLaunchCluster.py", line 73, in main
    keyName=config.keyPairName, spotBid=spotBid, userTags=tagsDict, zone=config.zone)
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/toil-3.7.0a1-py2.7.egg/toil/provisioners/aws/awsProvisioner.py", line 361, in launchCluster
    spec=kwargs, num_instances=1)
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/cgcloud_lib-1.6.0-py2.7.egg/cgcloud/lib/ec2.py", line 416, in create_ondemand_instances
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/home/chapmanb/install/bcbio-vm/anaconda/lib/python2.7/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>VPCResourceNotSpecified</Code><Message>The specified instance type can only be used in a VPC. A subnet ID or network interface ID is required to carry out the request.</Message></Error></Errors><RequestID>513f943f-a03d-44c4-8790-4acd4dc6d29f</RequestID></Response>
```
I'm not sure if this is an AMI choice issue or something incorrect in my setup that we're not launching these inside a VPC. I didn't see an option for specifying a VPC. Thanks for any tips/tricks.
Following up on my own post, I dug into the code and realized it assumes a default VPC. I have an older AWS account without default VPCs, so I will submit a PR to allow users to supply a VPC subnet ID to create instances in.

With that in place, I can get further and start towards launching (still stuck on waiting for Docker but will debug that separately).

On topic for this thread, why is CoreOS chosen here? The manual click through licensing is an extra annoying step if you like automation, and the region availability seems problematic. Could we accomplish what you need for Toil with a standard Ubuntu image which is replicated across all the regions and has consistent and long term availability? I've had good luck with these: http://cloud-images.ubuntu.com/locator/ec2/
@chapmanb CoreOS has Docker pre-installed, which we need for the provisioner since it is based on the Toil/Mesos Docker container. The licensing is once per account so it shouldn't bother you again. 

I'm looking into the AMI problem, it is still showing up on the UCSC account but I can't see it on my private account.
I'm not sure what is going on here - I found a matching AMI but it has the wrong 'Owner' attribute. I don't know why a public AMI's visibility would be affected by different accounts. 

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -111,13 +111,14 @@ def _remainingBillingInterval(self, instance):
     @memoize
     def _discoverAMI(cls, ctx):
         def descriptionMatches(ami):
-            return ami.description is not None and 'stable 1068.9.0' in ami.description
+            return ami.description is not None and 'stable 1235.4.0' in ami.description
         coreOSAMI = os.environ.get('TOIL_AWS_AMI')
         if coreOSAMI is not None:
             return coreOSAMI
         # that ownerID corresponds to coreOS
         coreOSAMI = [ami for ami in ctx.ec2.get_all_images(owners=['679593333241']) if
                      descriptionMatches(ami)]
+        logger.debug('Found the following matching AMIs: %s', coreOSAMI)
         assert len(coreOSAMI) == 1
         return coreOSAMI.pop().id
 
@@ -281,7 +282,7 @@ def _waitForDockerDaemon(cls, ip_address):
         while True:
             output = cls._sshInstance(ip_address, '/usr/bin/ps', 'aux')
             time.sleep(5)
-            if 'docker daemon' in output:
+            if 'dockerd' in output:
                 # docker daemon has started
                 break
             else:


Test patch: