Repository: DataBiosphere/toil

Issue:
AWS marketplace product not available
Hi everyone,
I have some trouble launching a cluster on AWS. I followed the setup instructions in the docs and wanted to launch a cluster using:

`toil launch-cluster testcluster --leaderNodeType t2.medium --zone eu-central-1a --keyPairName id_rsa`

this is the output:

```
INFO:toil:Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
INFO:toil:Using default docker name of toil as TOIL_DOCKER_NAME is not set.
INFO:toil:Using default docker appliance of quay.io/ucsc_cgl/toil:4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe-py3.6 as TOIL_APPLIANCE_SELF is not set.
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:botocore.credentials:Found credentials in shared credentials file: ~/.aws/credentials
INFO:toil:Using default user-defined custom docker init command of  as TOIL_CUSTOM_DOCKER_INIT_COMMAND is not set.
INFO:toil:Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
INFO:toil:Using default docker name of toil as TOIL_DOCKER_NAME is not set.
INFO:toil:Using default docker appliance of quay.io/ucsc_cgl/toil:4.1.0-5ad5e77d98e1456b4f70f5b00e688a43cdce2ebe-py3.6 as TOIL_APPLIANCE_SELF is not set.
INFO:toil.lib.ec2:Creating t2.medium instance(s) ... 
Traceback (most recent call last):
  File "/home/jaro/myvenv/bin/toil", line 11, in <module>
    sys.exit(main())
  File "/home/jaro/myvenv/lib/python3.6/site-packages/toil/utils/toilMain.py", line 25, in main
    module.main()
  File "/home/jaro/myvenv/lib/python3.6/site-packages/toil/utils/toilLaunchCluster.py", line 154, in main
    awsEc2ExtraSecurityGroupIds=config.awsEc2ExtraSecurityGroupIds)
  File "/home/jaro/myvenv/lib/python3.6/site-packages/toil/provisioners/aws/awsProvisioner.py", line 172, in launchCluster
    spec=specKwargs, num_instances=1)
  File "/home/jaro/myvenv/lib/python3.6/site-packages/toil/lib/ec2.py", line 255, in create_ondemand_instances
    **spec).instances
  File "/home/jaro/myvenv/lib/python3.6/site-packages/boto/ec2/connection.py", line 977, in run_instances
    verb='POST')
  File "/home/jaro/myvenv/lib/python3.6/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 401 Unauthorized
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>OptInRequired</Code><Message>In order to use this AWS Marketplace product you need to accept terms and subscribe. To do so please visit https://aws.amazon.com/marketplace/pp?sku=ryg425ue2hwnsok9ccfastg4</Message></Error></Errors><RequestID>905b50f3-9e44-4dc3-befb-8ca28ed506a0</RequestID></Response>
```

It says that I have to subscribe to an AWS marketplace product, but when following the link it says that Container Linux by CoreOS (Stable) is no longer available to new customers.
Is there any way around this?
Thanks in advance,
Jaro

┆Issue is synchronized with this [Jira Task](https://ucsc-cgl.atlassian.net/browse/TOIL-551)
┆Issue Number: TOIL-551



Discussion:
Sounds like this is https://github.com/DataBiosphere/toil/issues/2954 coming to fruition. CoreOS has pulled its images (or the "marketplace" thing that lets you find the right image for your region) from AWS, since it isn't being updated anymore. Presumably they think we would prefer all published versions of Toil breaking to something being compromised.

We need to change this here to something that matches a compatible AMI that actually exists still:

https://github.com/DataBiosphere/toil/blob/ba1f150dbb2f28081227e9f0d52748928d0be76c/src/toil/provisioners/aws/awsProvisioner.py#L389

The workaround is to set `TOIL_AWS_AMI` to an AMI that still exists; either your own upload of CoreOS, or something compatible like Flatcar. [Look up the right AMI for your region here](https://docs.flatcar-linux.org/os/booting-on-ec2/#choosing-a-channel) and try using that:

```
TOIL_AWS_AMI=ami-0a9a5d2b65cce04eb toil launch-cluster testcluster --leaderNodeType t2.medium --zone eu-central-1a --keyPairName id_rsa
```
I did some testing today; people on Amazon accounts that have already used Toil to start a cluster (and thus have the CoreOS marketplace entry hooked up) can still do so, but AWS accounts that haven't already used this marketplace product can't do it now. We either need to make our own marketplace product or tell Toil how to find the right AMI for every AWS region (at which point we maybe might as well upgrade to Flatcar).
Looks like flatcar use is now urgent.  Guess we should add it to the sprint.

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -17,6 +17,8 @@
 from builtins import range
 import time
 import string
+import json
+import urllib.request
 
 # Python 3 compatibility imports
 from _ssl import SSLError
@@ -376,22 +378,52 @@ def _buildContext(self):
 
     @memoize
     def _discoverAMI(self):
-        def descriptionMatches(ami):
-            return ami.description is not None and 'stable 1855.5.0' in ami.description
-        coreOSAMI = os.environ.get('TOIL_AWS_AMI')
-        if coreOSAMI is not None:
-            return coreOSAMI
-
-        for attempt in retry(predicate=lambda e: isinstance(e, SSLError)):
-            # SSLError is thrown when get_all_images times out
+        """
+        :return: The AMI ID (a string like 'ami-0a9a5d2b65cce04eb') for CoreOS
+                 or a compatible replacement like Flatcar.
+        :rtype: str
+        """
+        
+        # Take a user override
+        ami = os.environ.get('TOIL_AWS_AMI')
+        if ami is not None:
+            return ami
+        
+        # CoreOS is dead, long live Flatcar
+        
+        # Flatcar images, however, only live for 9 months.
+        # Rather than hardcode a list of AMIs by region that will die, we use
+        # their JSON feed of the current ones. 
+        JSON_FEED_URL = 'https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_ami_all.json'
+        
+        # What region do we care about?
+        region = zoneToRegion(self._zone)
+        
+        for attempt in retry(predicate=lambda e: True):
+            # Until we get parseable JSON
+            # TODO: What errors do we get for timeout, JSON parse failure, etc?
             with attempt:
-                # 679593333241 is the aws marketplace account
-                amis = self._ctx.ec2.get_all_images(owners=['679593333241'], filters={'name': 'CoreOS-stable-1855.5.0-hvm-0d1e0bd0-eaea-4397-9a3a-c56f861d2a14-ami-0f74e41ea6c13f74b.4'})
-
-        coreOSAMI = [ami for ami in amis if descriptionMatches(ami)]
-        logger.debug('Found the following matching AMIs: %s', coreOSAMI)
-        assert len(coreOSAMI) == 1, coreOSAMI
-        return coreOSAMI.pop().id
+                # Try to get the JSON and parse it.
+                feed = json.loads(urllib.request.urlopen(JSON_FEED_URL).read())
+                
+        try:
+            for ami_record in feed['amis']:
+                # Scan the klist of regions
+                if ami_record['name'] == region:
+                    # When we find ours
+                    # Save the AMI ID
+                    ami = ami_record['hvm']
+                    # And stop scanning
+                    break
+        except KeyError:
+            # We didn't see a field we need
+            raise RuntimeError('Flatcar image feed at {} does not have expected format'.format(JSON_FEED_URL))
+        
+        if ami is None:
+            # We didn't find it
+            raise RuntimeError('Flatcar image feed at {} does not have an image for region {}'.format(JSON_FEED_URL, region))
+
+        return ami
 
     def _toNameSpace(self):
         assert isinstance(self.clusterName, (str, bytes))
diff --git a/src/toil/provisioners/gceProvisioner.py b/src/toil/provisioners/gceProvisioner.py
--- a/src/toil/provisioners/gceProvisioner.py
+++ b/src/toil/provisioners/gceProvisioner.py
@@ -39,7 +39,7 @@ class GCEProvisioner(AbstractProvisioner):
     """
 
     NODE_BOTO_PATH = "/root/.boto" # boto file path on instances
-    SOURCE_IMAGE = (b'projects/coreos-cloud/global/images/family/coreos-stable')
+    SOURCE_IMAGE = (b'projects/flatcar-cloud/global/images/family/flatcar-stable')
 
     def __init__(self, clusterName, zone, nodeStorage, sseKey):
         super(GCEProvisioner, self).__init__(clusterName, zone, nodeStorage)
@@ -137,7 +137,7 @@ def launchCluster(self, leaderNodeType, leaderStorage, owner, **kwargs):
 
         userData =  self._getCloudConfigUserData('leader')
         metadata = {'items': [{'key': 'user-data', 'value': userData}]}
-        imageType = 'coreos-stable'
+        imageType = 'flatcar-stable'
         sa_scopes = [{'scopes': ['compute', 'storage-full']}]
         disk = {}
         disk['initializeParams'] = {
@@ -243,7 +243,7 @@ def addNodes(self, nodeType, numNodes, preemptable, spotBid=None):
         #kwargs["subnet_id"] = self.subnetID if self.subnetID else self._getClusterInstance(self.instanceMetaData).subnet_id
         userData =  self._getCloudConfigUserData('worker', self._masterPublicKey, keyPath, preemptable)
         metadata = {'items': [{'key': 'user-data', 'value': userData}]}
-        imageType = 'coreos-stable'
+        imageType = 'flatcar-stable'
         sa_scopes = [{'scopes': ['compute', 'storage-full']}]
         disk = {}
         disk['initializeParams'] = {


Test patch:
diff --git a/src/toil/test/provisioners/aws/awsProvisionerTest.py b/src/toil/test/provisioners/aws/awsProvisionerTest.py
--- a/src/toil/test/provisioners/aws/awsProvisionerTest.py
+++ b/src/toil/test/provisioners/aws/awsProvisionerTest.py
@@ -26,11 +26,23 @@
 
 import subprocess
 from toil.provisioners import clusterFactory
+from toil.provisioners.aws.awsProvisioner import AWSProvisioner
 from toil.version import exactPython
 from toil.test import needs_aws_ec2, integrative, ToilTest, needs_appliance, timeLimit, slow
 
 log = logging.getLogger(__name__)
 
+class AWSProvisionerBenchTest(ToilTest):
+    """
+    Tests for the AWS provisioner that don't actually provision anything.
+    """
+    
+    def testAMIFinding(self):
+        for zone in ['us-west-2a', 'eu-central-1a', 'sa-east-1b']:
+            provisioner = AWSProvisioner('fakename', zone, 10000, None)
+            ami = provisioner._discoverAMI()
+            # Make sure we got an AMI and it looks plausible
+            assert(ami.startswith('ami-'))
 
 @needs_aws_ec2
 @needs_appliance
@@ -410,3 +422,4 @@ def _runScript(self, toilOptions):
         command = ['/home/venv/bin/python', '/home/userScript.py']
         command.extend(toilOptions)
         self.sshUtil(command)
+