Repository: deis/deis

Issue:
Discussion: rewrite EC2 provision scripts (create a VPC for Deis, support T2/M3/C3 instance types, HVM instances)
**VPC**
Currently, our provision script for EC2 frequently runs into problems. We see weird errors like:

"Security group(s) can be applied to only an ELB in VPC" (#1207)

This may be caused by some accounts not having access to EC2-Classic, or some regions not having a default VPC. This is frustrating for users. Using a VPC seems to work more reliably, and should be the recommended way to deploy Deis on EC2.

The EC2 provision script should do one of two things:
1.  Create a VPC with sane defaults and provision Deis into it
2.  Provision Deis into an existing VPC by specifying `VPC_ID`, `VPC_SUBNETS`, `VPC_ZONES` (exactly as we do now)

This would enable power users who already have their own networks set up in VPC to provision Deis exactly as their use case dictates, but also enables new users to prototype Deis quickly on EC2.

Some discussion points:
- Are there any arguments against CloudFormation to deploy Deis (for either provision method), or should we start everything manually via the CLI? What are the pros and cons to this?
- What are sane VPC defaults? (Especially things like `AssociatePublicIpAddress` which could have security implications)

This is related to #1406 - some of this work may be moot if we go the route of a cloud-agnostic provisioner script.

**Arbitrary instance types**
We also need to rewrite the scripts to support arbitrary instance types and template CoreOS images for both PV and HHVM. We may be at the point where we give up on CloudFormation - we're using our gen-json.py script to override so many template options that it's getting hard to follow.



Discussion:
For the most pressing issue... It is fairly trivial to have CloudFormation build a new, and dedicated VPC for Deis. I'll throw something together and reference this issue in the PR.

I'd posit that if we want to make spinning Deis up on EC2 simple, making a foolproof CloudFormation template should be required. The defaults should give you a secure, and useable cluster without any intervention (except for DNS - which we could even change with CloudFormation). Allowing folks to customize the default CloudFormation template who have experience would be the best.

CloudFormation isn't a horrible solution for AWS. The current implementation initially gave us some grief with regard to VPC_ID being set but there not being a "default" VPC under the account (we had deleted the first VPC we created under our AWS account). This is a common AWS hurdle that any EC2Classic customers unfamiliar with VPCs will likely encounter. Unless everything has an explicit VPC_ID specified, the resource will attempt to use the default VPC. If you've deleted your account's default VPC, you must open a support ticket with AWS to add a VPC back as the "default" for your account. There is apparently no way to do this yourself.

We initially tried to add AssociatePublicIpAddresses for our testbed when the nodes wouldn't connect outbound to get anything. It is likely to be a common first hurdle for those first adopting Deis under AWS. Even with AssociatePublicIpAddresses enabled in the VPC subnet and in the CloudFormation definition, public IPs were still not being assigned. We ended up assigning elastic IPs to enable the nodes to connect outbound. We also ended up provisioning a "console" node using chef onto the same VPC segment so we could ssh to it and managed the Deis cluster there with http until we could get HTTPS to work. Ideally, the internet gateway allocated would DNAT all outbound internet traffic, and the Deis nodes would be protected from the public Internet ssh port scans (try enabling fail2ban on your coreos nodes otherwise).

The second hurdle was logging, but that was easily rectified within Deis itself and the addition of logspout docker containers on each node.

After deployment, our next major change was allowing HTTPS to the ELB security group (which should probably be the default), though adding an HTTPS listener to an ELB is made non-trivial by the fact that importing an SSL cert into the ELB requires manual intervention via the AWS console.

Beyond this, there are questions of network topology lockdown, sandboxing the app containers away from the Deis containers and the Deis nodes themselves, for example. While there might be reasons containers might like to use the etcd instances on the Deis nodes, that really belongs in a containerized layer, but there are still nagging questions of how to orchestrate rolls to retain quorum through that layer using the existing scheduler, etc. 

Network topology questions, node security group style lockdown questions, load balancing questions, provisioning bootstrap questions, scheduler elastic scaling questions... so many questions. Taken as a whole, this isn't a trivial problem. Perhaps the question should be: how do we slice this into simple abstractions that layer well and still allow servicing other clouds in a similar topology as per #1406?

Regarding the AMI/PV/HVM issue, we have the following facts:
1. Some instance types only support one type of virtualization but most instance types support both. Actually, it turns out that t1 (the old micro) instances only support PV while t2 (new micro) only support HVM.
2. t2 instances are important from a first time users perspective because they are included in the AWS free usage tier.
3. t1.micro instances are still available in the free usage tier and probably will be for at least a few months (years?)
4. We can expect better performance from PV so 'always use HVM' is not the best solution.
5. There are actually 'alpha' and 'beta' CoreOS AMIs and it looks like the alpha images are used at the moment for some arbitrary (to me) reason
6. After briefly skimming cloudformation documentation it looks like it's possible but slightly messy to support 'if t2 use HVM else use PV' type logic

So I think fact 3 makes this whole issue moot for now since people can just use t1.micros to get their free tier goodness, but in the longer run...

Should the script pick which AMI to use (alpha/beta, PV/HVM), make the user decide or have a default that can be modified?

One issue to keep in mind is that with CloudFormation/LaunchConfig you'll have to continually keep the AMI up to date with the latest CoreOS release. There isn't a way to reference a "latest" or "current" image with a CF/LC initially launched months earlier. This means that if your autoscaling group boots a new machine, it launches on the old AMI, then immediately updates (if configured) and then you're good to go after the reboot. Obviously this isn't ideal if you need to start using your machine right as it boots.

We've talked to some AWS engineers about this issue and there isn't an elegant solution. If you used a script instead, it could look up the latest AMI ID and use that.

I know you guys aren't enabling updates currently, but if thats one your roadmap, this is something to be aware of.

@robszumski Good to know - thanks! We're currently updating the CloudFormation manually, but automatic updates with locksmithd is something we're closing in on.


Resolving patch:
diff --git a/contrib/ec2/gen-json.py b/contrib/ec2/gen-json.py
--- a/contrib/ec2/gen-json.py
+++ b/contrib/ec2/gen-json.py
@@ -14,24 +14,27 @@
 VPC_SUBNETS = os.getenv('VPC_SUBNETS', None)
 VPC_ZONES = os.getenv('VPC_ZONES', None)
 
-if VPC_ID and VPC_SUBNETS and VPC_ZONES:
-  for resource in template['Resources'].keys():
-    resource_type = template['Resources'][resource]['Type']
-    if resource_type == 'AWS::EC2::SecurityGroup':
-      template['Resources'][resource]['Properties']['VpcId'] = VPC_ID
-    elif resource_type == 'AWS::EC2::SecurityGroupIngress':
-      template['Resources'][resource]['Properties']['GroupId'] = template['Resources'][resource]['Properties']['GroupName']
-      del template['Resources'][resource]['Properties']['GroupName']
-      template['Resources'][resource]['Properties']['SourceSecurityGroupId'] = {
-        'Ref': template['Resources'][resource]['Properties']['SourceSecurityGroupId']['Fn::GetAtt'][0]
-      }
-    elif resource_type == 'AWS::AutoScaling::LaunchConfiguration':
-      template['Resources'][resource]['Properties']['AssociatePublicIpAddress'] = False
-    elif resource_type == 'AWS::ElasticLoadBalancing::LoadBalancer':
-      del template['Resources'][resource]['Properties']['AvailabilityZones']
-      template['Resources'][resource]['Properties']['Subnets'] = VPC_SUBNETS.split(',')
-    elif resource_type == 'AWS::AutoScaling::AutoScalingGroup':
-      template['Resources'][resource]['Properties']['VPCZoneIdentifier'] = VPC_SUBNETS.split(',')
-      template['Resources'][resource]['Properties']['AvailabilityZones'] = VPC_ZONES.split(',')
+if VPC_ID and VPC_SUBNETS and VPC_ZONES and len(VPC_SUBNETS.split(',')) == len(VPC_ZONES.split(',')):
+  # skip VPC, subnet, route, and internet gateway creation
+  del template['Resources']['VPC']
+  del template['Resources']['Subnet1']
+  del template['Resources']['Subnet2']
+  del template['Resources']['Subnet1RouteTableAssociation']
+  del template['Resources']['Subnet2RouteTableAssociation']
+  del template['Resources']['InternetGateway']
+  del template['Resources']['GatewayToInternet']
+  del template['Resources']['PublicRouteTable']
+  del template['Resources']['PublicRoute']
+  del template['Resources']['CoreOSServerLaunchConfig']['DependsOn']
+  del template['Resources']['DeisWebELB']['DependsOn']
+
+  # update VpcId fields
+  template['Resources']['DeisWebELBSecurityGroup']['Properties']['VpcId'] = VPC_ID
+  template['Resources']['VPCSecurityGroup']['Properties']['VpcId'] = VPC_ID
+
+  # update subnets and zones
+  template['Resources']['CoreOSServerAutoScale']['Properties']['AvailabilityZones'] = VPC_ZONES.split(',')
+  template['Resources']['CoreOSServerAutoScale']['Properties']['VPCZoneIdentifier'] = VPC_SUBNETS.split(',')
+  template['Resources']['DeisWebELB']['Properties']['Subnets'] = VPC_SUBNETS.split(',')
 
 print json.dumps(template)


Test patch: