Change:
return fmt.Errorf("rbd: failed to modprobe rbd error:%v", err) } // rbd map <del> l := len(rbd.mon) </del> <ins> l := len(rbd.Mon) </ins> // avoid mount storm, pick a host randomly start := rand.Int() % l // iterate all hosts until mount succeeds. for i := start; i < start+l; i++ { <del> mon := rbd.mon[i%l] </del> <ins> mon := rbd.Mon[i%l] </ins> glog.V(1).Infof("rbd: map mon %s", mon) <del> if rbd.secret != "" { </del> <ins> if rbd.Secret != "" { </ins> _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "--key=" + rbd.secret}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "--key=" + rbd.Secret}) </ins> } else { _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "-k", rbd.keyring}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "-k", rbd.Keyring}) </ins> } if err == nil { break

Discussion:
etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase.

, Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.