facebook/flashcache — Issue #54: flashcache + drbd + two node kvm cluster -> VM image corruption after live migration
Author: MvdLande
Created: 2012-02-24 09:31:18+00:00
State: open

Hello,

I'm testing a configuration with flashcache on a two node kvm cluster. I have setup these two node clusters before without flashcache without any trouble. When I use flashcache then I experience VM image file corruption after using virsh live migration. For some reason the data on both kvm hosts is not the same. I have tested both write through and write back mode with the same result.

Both the flashcache and the file storage are using disk partitions, could this be a problem?

Problem description:
When using flachcache + drbd in the configuration mentioned below I get a corrupted KVM Virtual Machine Image file after using virsh live migration. This is noticed after repeatedly live migrating the VM between two host servers. The VM host OS is Windows 7 x86 . Without flashcache I get no corrupted VM image file (tested 2000+ live migrations on the same VM) I'm also using the latest virtio-win drivers.

Host OS = Centos 6.2 x86_64
The migration is started using the following command
# virsh migrate --live --verbose test qemu+ssh://vmhost2a.vdl-fittings.local/system
And back again with after 5 minutes
# virsh migrate --live --verbose test qemu+ssh://vmhost2b.vdl-fittings.local/system
(I used cron to automate the task so the VM is live migrated every 5 minutes)

Symptom:
The VM Image corruption was noticed because windows began complaining about unreadable/corrupted system files. Also after rebooting the VM numerous errors were detected by windows disk check. It looks like the cached data is not the same on both servers. I have configured drbd to use the flashcache device so all drbd data should pass through the cache. Also static files became unreadable after a while. (when I tried to open a folder with some files, the folder was corrupted)

Configuration:
The two host servers are using GFS2 as the cluster storage file system to host the Image files. (which works fine without flashcache)

I have the following disk setup:
/dev/sda1	raid1 SSD array (200GB) (using an Adaptec 6805 controller)
/dev/sdc1	raid5 HD array (1.5TB) (using another Adaptec 6805 controller)
/dev/mapper/cachedev	the flashcache device

As for flashcache, I have used two setups
- 1 using flashcache in write through mode
# /sbin/flashcache_create -p thru -b 16k cachedev /dev/sda1 /dev/sdc1
- 2 using flashcache in write back mode
# /sbin/flashcache_create -p back -b 16k cachedev /dev/sda1 /dev/sdc1
Both experienced the same VM Image corruption.

The disk /dev/drbd0 is mounted on /VM

My drbd setup is as follows: (using drbd8.3.12)
#------------------------------------------------------------------------
# Just for testing I kept all configuration in one file #include "drbd.d/global_common.conf"; #include "drbd.d/*.res"; global {
minor-count 64;
usage-count yes;
}

common {
syncer {
rate 110M;
verify-alg sha1;
csums-alg sha1;
al-extents 3733;
# cpu-mask 3;
}
}

resource VMstore1 {

protocol C;

startup {
wfc-timeout 1800; # 30 min
degr-wfc-timeout 120; # 2 minutes.
wait-after-sb;
become-primary-on both;
}

disk {
no-disk-barrier;
# no-disk-flushes;
}

net {
max-buffers 8000;
max-epoch-size 8000;
sndbuf-size 0;
allow-two-primaries;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
}
syncer{
cpu-mask 3;
}

on vmhost2a.vdl-fittings.local {
device /dev/drbd0;
disk /dev/mapper/cachedev;
address 192.168.100.3:7788;
meta-disk internal;
}
on vmhost2b.vdl-fittings.local {
device /dev/drbd0;
disk /dev/mapper/cachedev;
address 192.168.100.4:7788;
meta-disk internal;
}
}
#------------------------------------------------------------------------

Cluster configuration: (no fence devices)
#------------------------------------------------------------------------

#------------------------------------------------------------------------
Flashcache version
#------------------------------------------------------------------------
[root@vmhost2a ~]# modinfo flashcache
filename: /lib/modules/2.6.32-220.4.2.el6.x86_64/weak-updates/flashcache/flashcache.ko
license: GPL
author: Mohan - based on code by Ming
description: device-mapper Facebook flash cache target
srcversion: E1A5D9AA620A2EACC9FA891
depends: dm-mod
vermagic: 2.6.32-220.el6.x86_64 SMP mod_unload modversions
#------------------------------------------------------------------------

libvirt xmldump VM configuration
#------------------------------------------------------------------------

- test a84df1cb-668f-cf5b-7433-91f99ac23971 2097152 2097152 4 - hvm - - Penryn Intel destroy restart restart - /usr/libexec/qemu-kvm -

-

-

- - - -

-

#------------------------------------------------------------------------