wr0ngway/rubber — pull request #127 by akremer (2012-02-11 04:56:29+00:00)
Title: RAID Fixes

This commit addresses a whole range of issues with RAID/mdadm in Ubuntu 10.x. It also adds compatibility for RAID in Ubuntu 11+.

Among the fixes:

- RAID volumes created by rubber could brick an instance on reboot due to the way RAID was configured, including trying to scan all volumes that would sometimes never be ready. This would prevent SSH from starting up since Ubuntu (for some reason) tells SSH to start only when all filesystems are ready. More info in the following links.

EC2 Ubuntu Raid0 Ephemeral - Reboot causes connection refused
http://stackoverflow.com/questions/8530691/ec2-ubuntu-raid0-ephemeral-reboot-causes-connection-refused

mdadm: breaks boot due to “is not ready yet or not present” error
http://askubuntu.com/questions/20857/mdadm-breaks-boot-due-to-is-not-ready-yet-or-not-present-error

mdadm not surviving reboot
http://ubuntuforums.org/printthread.php?t=1207071&pp=75

raid array does not auto start/mount
http://ubuntuforums.org/printthread.php?t=1468064&pp=75

The changes include adding "nobootwait" instructions in rubber.yml, only letting mdadm scan devices configured for raid in rubber.yml, and running update-initramfs after the RAID creation.

- RAID volumes now work in Natty and above. This was broken due to the /dev/sdX -> /dev/xvdX kernel change. The mdadm in Natty and above also had another bug that necessitated removing the "name" for each array in mdadm.conf, or it would (once again) brick SSH on reboot.

if array is given a name, a strange inactive md device appears instead of the one created upon reboot
https://bugs.launchpad.net/ubuntu/+source/mdadm/+bug/576147

- Backed out my previous commit to wait an additional few seconds post-EBS attachment and included a much more reliable fix to ensure volumes are ready before mdadm --create is called.

Best way to know EBS volume is attached?
https://forums.aws.amazon.com/thread.jspa?threadID=30362&tstart=0

- Creating RAID volumes is now idempotent. This wasn't the case because the script added the volume to /etc/fstab first thing before mdadm evan got a chance to run.

***

I've tested all of this on 32bit and 64bit instances of Lucid, Natty and Oneiric, but I'd encourage some more testing. There is a notable issue I haven't been able to solve where RAID volumes on Micro instances running Lucid still won't auto-mount when mountall is called on boot. This seems to be caused by the lack of ephemeral storage in Micro instances. Natty+ doesn't seem to exhibit this issue.