facebook/flashcache — Issue #39: I/O Freeze while md checking or rebuilding RAID volume
Author: tony3
Created: 2011-11-06 08:07:52+00:00
State: open

Just ran head-first into a bug where I lost all I/O for at least 10 minutes before I gave up (rebooted both machines with echo 1 > /proc/sys/kernel/sysrq; echo b > /proc/sysrq-trigger because 'reboot' just blocked waiting for the md device that root is on. Good thing sshd wasn't paged.)

It seems like this only happens on high disk i/o as I hung disk i/o both times while doing a 500Mbps rsync between two machines. It didn't hang the machines until a few tens of seconds into the rsync. First, the machines were doing an md check and then they were doing a rebuild after the first hang. When I removed the fastcache devices and accessed the md devices directly, the rsync and md resync ran fine concurrently at basically the same speed. Both machines crashed in the same way at basically the same time. FC is caching a 2-drive RAID1 md device with an OCZ AGILITY3 on each machine.

cat /proc/iostats showed no change in any field over the 10-minutes the machines hung except for fields 10 and 11 on the LVM devices on top of the cached md device.
http://www.kernel.org/doc/Documentation/iostats.txt

Kernel: (from Ubuntu 11.10)
3.0.0-12-server #20-Ubuntu SMP Fri Oct 7 16:36:30 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux

FC: (git head from 2011-11-02)
git commit: 1.0-121-g20e567f48970