Kabam/chef-mongodb — pull request #3 by mkantor (2013-10-31 01:35:06+00:00)
Title: EBS and snapshot support.

This pull request isn't intended to be merged; it's just here to get feedback. I plan on splitting these recipes out into a separate cookbook so that we don't diverge too far from upstream, but I wanted to get some review ASAP.

When reviewing, you can ignore files/default/ec2-consistent-snapshot. It's [a third-party project](https://github.com/alestic/ec2-consistent-snapshot). I may switch this to be a submodule in the future.

See also: .

## Relevant PT Tickets

- https://www.pivotaltracker.com/story/show/58097784
- https://www.pivotaltracker.com/story/show/54220252
- https://www.pivotaltracker.com/story/show/58098276
- https://www.pivotaltracker.com/story/show/58850770

## Changes

#### New Recipes
- `ebs_data`

Creates RAIDed EBS volumes and mounts them at `node[:mongodb][:dbpath]`. Currently this is hardcoded to use RAID 0. If the `node[:mongodb][:ebs][:create_from_snapshots]` attribute is `true`, the recipe will search for appropriate snapshots from which to initialize its EBS volumes.

- `ebs_snapshots`

Sets up a cron job to create snapshots of the mounted EBS volumes. This recipe must occur after `ebs_data` in run lists, as the volumes need to exist in order to configure snapshotting.

#### New Attributes

A number of new attributes have been added. Some of the interesting ones are listed below:

- `node[:mongodb][:setra]`

Configures disk read-ahead. This isn't specific to EBS, but is important there as the defaults are not optimized for MongoDB. See .

- `node[:mongodb][:ebs][...]`

Configure various aspects of the EBS volumes. See attributes/default.rb for more information.

- `node[:mongodb][:ebs][:create_from_snapshots]`

Whether to search for snapshots when creating EBS volumes or always start fresh.

- `node[:mongodb][:ebs][:snapshot_region]`

Region where snapshots will live. Eventually I plan on trying to make this automatically determined via [the EC2 instance metadata service](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html).

- `node[:mongodb][:ebs][:snapshot_cron_hour]` & `node[:mongodb][:ebs][:snapshot_cron_minute]`

Used to schedule snapshotting.

These changes are not documented in the README in this repo. I'll write up a proper README when this moves to its own cookbook.

Currently, AWS credentials need to be stored in an encrypted databag for this to work. My credentials are on the AWS chef server in the appropriate databag, so you shouldn't need to do anything special when testing. Unfortunately, the AWS cookbook that this depends on doesn't support IAM roles directly, but my eventual plan is to write another recipe to fetch temporary tokens via [STS](http://docs.aws.amazon.com/STS/latest/UsingSTS/Welcome.html).

## Testing

1. Clone the [the mongo-snapshots branch of distribution-devops](https://github.com/Kabam/distribution-devops/tree/mongo-snapshots).
2. Uniqueify the mongo role names in the CF templates (you can use *-mk; I've already uploaded roles with that suffix to the AWS chef server).
3. Start stacks using one of the customized mongo templates.
4. Wait...
5. Once the stack is ready, make sure EBS volumes have been properly mounted, and kill servers in creative ways to make sure they heal properly.

To test snapshots, you'll want to add some data. I put together a simple chef recipe to set up a cron job that inserts a timestamp into the DB every minute, you might find it useful:

```ruby
# insert the current date/time into "testCollection"
file "/tmp/mongo-test.js" do
content "db.testData.insert({ date: new Date() })"
owner "root"
mode 00777
end

# insert once per minute
cron "mongo_test_data" do
user "root"
minute "*"
command "mongo /tmp/mongo-test.js"
action :create
end
```

Of course, it's pretty easy to do this manually too.

Snapshotting occurs by default once every two hours. You can SSH in to the shapshooter node and run the `mongo-snapshot` script to create one at any time (it should be in the `$PATH`).

I have seen occasional cases where replacement secondaries trying to join the replica set take a long time to get initialized and have the message "replSet can't get local.system.replset config from self or any seed (EMPTYCONFIG)" repeated in their logs. I haven't been able to pin down what causes this yet. Eventually they are able to join.

Note: I haven't actually tested this with the full sharded cluster yet. Give it a go and let me know how spectacularly it breaks! If you want to do this, make sure to give the replica set template a custom name before uploading it to S3.

---

If you made it all the way through that wall of text, you deserve a cookie: :cookie:.