azavea/nyc-trees — pull request #67 by hectcastro (2014-11-07 18:54:25+00:00)
Title: Add Packer support

This changeset adds support for creating Amazon Machine Images (AMIs) that correspond to the types of Vagrant box definitions used in development. Because we will be making use of the Relational Database
Service (RDS) and ElastiCache (for Redis) in production, services like PostgreSQL and Redis are excluded from the images.

The 3 AMI types are:

- Monitoring

This AMI contains Kibana, Logstash, and Graphite/Statsite. The goal is to use this AMI for the bastion host, giving it two roles (bastion and monitoring server).

- Application

This AMI houses the Django application and maintains pointers to all of the services it depends on (tile, cache, database).

- Tile

This AMI contains the tiler and maintains pointers to the cache and database services.

---

Two other notable additions that were required for Packer support are private DNS support and a deployment helper to string together CloudFormation inputs/outputs.

- Private DNS support

Route 53 has support for private DNS. The plan is to use the `nyc-trees.internal` domain for services available within the Virtual Private Cloud (VPC). For example, the application server will always be
configured to talk to the cache via `cache.services.nyc-trees.internal`. This allows AMI creation to be more flexible in the event that we have to replace the caching cluster.

- Deployment helper

`deployment-helper.sh` contains almost all of the code to connect CloudFormation stacks. It is a fair amount of Bash, but it defers a lot of the heavy lifting to the `awscli`.

I originally started down the path of writing tooling in Python with Boto and Cement, but ran into hurdles due to my Python rustiness and Cement framework confusion. Also, I don't think that my Python code would have turned out better than the Python code in the `awscli`.

It would be great to revisit the CloudFormation tooling at a later point in a way that solves this problem across all Azavea teams.

See the `README` for more details on how an actual deployment would work.