liftopia/chef-interview — pull request #2 by jamesbrink (2014-04-11 09:25:27+00:00)
Title: James Brink

A friend of mine (@doomspork) mentioned you were looking for DevOps engineers so when I came across the chef-interview repo I knew I had to give it a shot. I have a number of years of experience managing and maintaining server infrastructures and I passion for solving complicated problems, I think I’d be a fit and valuable resource. I’d love to talk with someone if you have time.

Per the README here our my notes to accompany my work:

I started by firing up the Vagrant box to see what the state of the machine was after chef ran. My focus was on working from the most important issues down to the least. As of this writing I have been working with Chef for a little over a week but given the opportunity I could become very proficient with it.

The first thing I did was update the Chef recipes to use a shared pids directory for each application in `{{app-root}}/shared/tmp/pids` because the directory was referenced with symlinks but was missing. Next I altered the restart command to no longer kill the Ruby process and updated the recipe to use a switch to drop the pid file into the shared directory. This allows for a cleaner application restarts by iterating over any pid files in the shared directory.

My next step was to look into the issue of the application being re-deploy on provision, even though there had been no changes to the application. After looking at the documentation for the resource `deploy` and the directories on the box it was apparent the correct Chef resource was not being accessed. As you know the default behavior for `deploy` is to do a Capistrano style deployment using a timestamp. To fix the redeployment issue I switched to using `deploy_revision` which relies on the commit hash instead.

After both applications were deploying properly, I moved on to updating the recipes to deploy the application as user “liftopian” which I saw had been created in the recipe but not used.

My next set of changes included some refactoring of the application recipes. I took packages and services I felt could be re-used and turned them into cookbooks such as: Ruby, MongoDB, NodeJS, and Redis. While these cookbooks are simple and made to run only on Ubuntu, doing this will allow for re-use of these common packages and services. If at some point the infrastructure included hosts other than Ubuntu, it might be worthwhile to leverage the community cookbooks. Extracting these out will allow greater control down the road and separation of concerns.

My last step was to ensure that the Alpha and Beta app would start on boot. I noticed there was a cookbook for runit which I was not familiar with nor am sure how people were using it.

After reading a bit of information on the project’s homepage it I realized it was a replacement for upstart or systemd and while I think including a process/service supervisor is a great idea I was turned off by the fact they state the following.

“The program runit is intended to run as Unix process no 1, it is automatically started by the runit-init /sbin/init-replacement if this is started by the kernel.”

I feel if runit is used in this capacity it could be problematic especially if you intent to be flexible enough to provision other hosts such as FreeBSD or Solaris. With the wide variety of init daemons out there right now a process supervisor is great to have, so you don’t spend time writing upstart, systemd, init, or SMF startup scripts. I ended up choosing Supervisor as it is clean, simple, and works across platforms. What really caught my eye about Supervisor beyond its features, was that it is __not__ intended to be a placement for init.

Prior to this exercise I had never used Supervisor but I must say I am very impressed with the simplicity of the tool, and will likely use it again; I went ahead and created a chef cookbook for it. I also created a template to ensure that the user “liftopian” would be able to start and stop processes, I updated the application recipes to require in this new recipe and dropped in the configuration file for each.

After everything was running, I took a look at the applications Gemfiles and noticed the unicorn gem. I decided to assume this is a production box and altered the application recipes to use Unicorn along with nginx.

Next Steps/Conclusion:

The next steps I would take would be to either expand the basic cookbooks I created or leverage a community cookbook. I also noted that gems were installed to the `/tmp/` directory, in general this is a bad location because it consumes your swap and will not survive reboots, I change it to `/var/tmp/` but it really should have a new home. If this were true production we would also need to profile the cpu and memory utilization and adjust workers and configs for optimal performance. Additional work could be done to create 2 environments for Production and Development.

A final note, I did use a lot of small commits to help illustrate my workflow.

Thanks for you time.
James Brink