scraperwiki/swops — pull request #7 by pwaller (2013-11-17 19:20:44+00:00)
Title: Add clean-snapshots.py script

CC @drj11 @morty @frabcus

Currently snapshots are costing us around $200/mo and just generally there
are many more than we need. Also I'm trying to cull the swops issues, so I
slapped something together to kill unneeded snapshots.

xref #1

![image](https://f.cloud.github.com/assets/438648/1559025/e9dbf6cc-4fba-11e3-8c27-af8b7171a693.png)

Adds a script called clean-snapshots.txt which takes in a list of snapshots
from `aws ec2 describe-snapshots` and emits a list of snapshot IDs which can
be deleted. I also used python's `calendar` module to observe that the correct
images are being retained.

[The core of the logic is here](https://github.com/scraperwiki/swops/blob/16f99d2f4dd2f12253e5eee25cb8c6fdcd1d6f20/ec2-backup/snapshots/clean-snapshots.py#L141).

The intent is to, for each machine: A) keep the latest ten snapshots, B) keep snapshots younger than two weeks C) Keep saturday snapshots less than 6 weeks, D) keep every other saturday (based on week number of year % 2) E) keep every fourth snapshot (again based on week number % 4.).

It assumes that there is a snapshot every day. I chose Saturday on the grounds
that not much changes at the weekends, so it may be an "isle of stability".

It uses the `Name` tag of AWS machines to bucket snapshots into machines, using
`rpartition(':')` to remove the date on the right hand side of the name, or if
there isn't a right hand side, it uses the full name of the image. Then the
above rules are applied. That means that if you make a snapshot according to
some other name, it will have the same convention applied
("keep the last ten shapshots", etc).

At the moment the output of the script shows a calendar for each "machine" and
tells the user how many snapshots there are and how many would be deleted and
kept. It's up to the user to go and run the `aws --region=[region] delete-snapshot --snapshot-id [snapshot-id]` for themselves. Since this is a slow-moving problem
I it doesn't seem worth automating for now and also gives a chance to check
over the snapshots.