Repository: DataBiosphere/toil

Issue:
Toil launch-cluster exceeded the number of VPC security groups allowed per instance
```
$ toil launch-cluster spark-toil \
  --nodeType=t2.micro \
  -z us-east-1e \
  --vpc vpc-foo \
  --vpcSubnet subnet-bar \
  --keyPairName=baz

2017-06-08 13:53:17,386 MainThread INFO toil.utils.toilLaunchCluster: Using aws provisioner.
2017-06-08 13:53:19,595 MainThread INFO toil: Using default docker registry of quay.io/ucsc_cgl as TOIL_DOCKER_REGISTRY is not set.
2017-06-08 13:53:19,595 MainThread INFO toil: Using default docker name of toil as TOIL_DOCKER_NAME is not set.
2017-06-08 13:53:19,596 MainThread INFO toil: Using default docker appliance of quay.io/ucsc_cgl/toil:3.7.0-73c62d7698bb6f4154d276994df4353ce4d7826b as TOIL_APPLIANCE_SELF is not set.
2017-06-08 13:53:19,596 MainThread INFO toil.provisioners.aws.awsProvisioner: Launching non-preemptable leader
2017-06-08 13:54:04,223 MainThread INFO cgcloud.lib.ec2: Creating t2.micro instance(s) ... 
Traceback (most recent call last):
  File "/usr/local/bin/toil", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilMain.py", line 33, in main
    module.main()
  File "/usr/local/lib/python2.7/site-packages/toil/utils/toilLaunchCluster.py", line 77, in main
    vpcSubnet=config.vpcSubnet)
  File "/usr/local/lib/python2.7/site-packages/toil/provisioners/aws/awsProvisioner.py", line 398, in launchCluster
    spec=kwargs, num_instances=1)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/cgcloud/lib/ec2.py", line 414, in create_ondemand_instances
    **spec ).instances
  File "/usr/local/lib/python2.7/site-packages/boto/ec2/connection.py", line 973, in run_instances
    verb='POST')
  File "/usr/local/lib/python2.7/site-packages/boto/connection.py", line 1208, in get_object
    raise self.ResponseError(response.status, response.reason, body)
boto.exception.EC2ResponseError: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response>
  <Errors>
    <Error>
      <Code>SecurityGroupLimitExceeded</Code>
      <Message>You have exceeded the number of VPC security groups allowed per instance.</Message>
    </Error>
  </Errors>
  <RequestID>47f1ee07-0257-4338-b546-4a2a81582fd6</RequestID>
</Response>
```
Found this, which may be related, except I would imagine mine is an old account, not a new one
https://github.com/BD2KGenomics/toil/pull/1512#issuecomment-280105381

┆Issue is synchronized with this [JIRA Story](https://ucsc-cgl.atlassian.net/browse/TOIL-210)
┆Issue Number: TOIL-210



Discussion:
The limit seems to be 500.
http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits.html

Maybe Toil is creating unnecessary groups.
When this happened the first time, the instance was still up, and I could see it in the EC Management Console.  The core-os image adds a bunch of security groups in addition to those added by Toil.

Unfortunately when I reproduce the error again today, I'm not seeing the instance still up in the EC2 console to check the security groups.

I'm using toil 3.7.0 and the default appliance image (`Using default docker appliance of quay.io/ucsc_cgl/toil:3.7.0-73c62d7698bb6f4154d276994df4353ce4d7826b as TOIL_APPLIANCE_SELF is not set.`); looks like there has been a lot going on since, should I try more recent versions of either?
If it is a problem with Toil, I don't think it has been addressed. I have seen a transient 400 boto error. It could be related.

Is this a transient error? Are you blocked on this?
I am running into this, with latest Toil, anyone know why this might happen?
@tom-dyar The collaborator I was working with wasn't able to identify the issue nor configure AWS to prevent the error, so we gave up on Toil provisioning and used Elastic Map Reduce (EMR) clusters instead.

What is "DataBiosphere" by the way?
@heuermh Not sure about "DataBiosphere" ;)

re. EMR clusters, I would be interested in that -- is that laid out in the documentation somewhere??

re. security groups, I can see that the provisioner creates its own security group for the cluster, but also grabs and tries to apply all the security groups (in the vpc/subnet) and sends those to EC2 as part of the "spec" for creating the instance. In my case, there are about 10 security groups for other apps and stuff "laying around", so that blows up the limit of 5 security groups per instance. I guess that code assumes all the security groups are relevant and should be applied, so *maybe* I can pare down the list via some argument or config parameter somewhere??
OK, @chapmanb, @ejacox looks like this code might be missing parens in the conditional, starting line 550 in src/toil/provisioners/aws/awsProvisioner.py:

        out = []
        for sg in self._ctx.ec2.get_all_security_groups():
            if sg.name == self.clusterName and vpcId is None or sg.vpc_id == vpcId:
                out.append(sg)
        return out

That code may not do what it is intended, since in my account, all the sg's in the current VPC get appended. Should it be more like this?:
...
if sg.name == self.clusterName and (vpcId is None or sg.vpc_id == vpcId):
...

Thanks for looking at this!

Thomas -- thanks for the detective work. I'm agreed with your fix, this does look like a logic bug in my code. I haven't gotten to use the AWS setup in a while but would be +1 for this change based on reading the code. Thanks again.

Resolving patch:
diff --git a/src/toil/provisioners/aws/awsProvisioner.py b/src/toil/provisioners/aws/awsProvisioner.py
--- a/src/toil/provisioners/aws/awsProvisioner.py
+++ b/src/toil/provisioners/aws/awsProvisioner.py
@@ -549,7 +549,7 @@ def groupNotFound(e):
                     web.authorize(ip_protocol='udp', from_port=0, to_port=65535, src_group=web)
         out = []
         for sg in self._ctx.ec2.get_all_security_groups():
-            if sg.name == self.clusterName and vpcId is None or sg.vpc_id == vpcId:
+            if sg.name == self.clusterName and (vpcId is None or sg.vpc_id == vpcId):
                 out.append(sg)
         return out
 


Test patch: