Change:
etcd cluster backing apiserver died spontaneously, after restart apiserver pulled up the pods on different machines. Node storage was backed by ext4 over rbd, which became mounted on several nodes at once. That could have possibly led to data corruption. I guess that the proper way to mount rbd volume is to take a lock on it first, which will at least fail if it's locked somewhere else, requiring manual intervention, but not destroying data. The lock should be optional, in case a proper cluster fs is mounted over the rbd. In this case there was clearly no need for accessing the data from several pods so ext4 is deemed a proper fs.
, Vladimir Pouzanov < :
want to get some detail. How is your pod-to-rbd mapping like? Since rbd image is a block device, it can be used by one pod at a time. I am not sure if locking is a proper way to go - if the pod holds a lock but dies in the middle of mounting, the rbd image (or any other volume type) is then excluded from access. In such case, a high level coordination at apiserver is a better place to resolve this problem.
Here's my rc: I guess it's better to have a lock to be cleaned up manually rather than to have non-cluster fs to be mounted in several locations at once?
thanks, this pod looks good to me. what would cause multiple nodes run the same pod?
I don't know the exact reason of why kubelet didn't unmount the old fs, but after apiserver and scheduler were restarted pod got allocated on different node.
It is a scary thing to see etcd down, it is meant to be reliable...
Unfortunately I don't have long-term logging in test cluster so I have no idea why it actually died. but using kubedns as primary node resolver was clearly a bad idea, once it died, nodes stuck with no resolver at all.
your thought on HA?
Wrong Tim St. Clair?
- you HAVE to be ready to handle network partitions - the old node might still be alive and even using the volume, but the apiserver can't see the node, so it thinks it deletes the pod and starts it elsewhere. If RBD corrupts itslef in this case, it is not useful and we should strip it from the codebase. , Huamin Chen wrote:
that makes sense. rbd doesn't corrupt itself. This is inconsistency happens when you have two pods mounting the same rbd image. How does GCE PD handle/prevent this case?
It doesn't allow more than one read-write attachment. Multiple read-only attachments is OK. Better to fail than to allow corruption. , Huamin Chen wrote:
That's neat. I have a lock fencing for rbd, testing now.

Discussion:
} // Perform a bind mount to the full path to allow duplicate mounts of the same disk. options := []string{"bind"} <del> if disk.readOnly { </del> <ins> if disk.ReadOnly { </ins> options = append(options, "ro") } err = mounter.Mount(globalPDPath, volPath, "", options)

return &rbd{ podUID:   podUID, volName:  spec.Name, <del> mon:      source.CephMonitors, image:    source.RBDImage, pool:     pool, id:       id, keyring:  keyring, secret:   secret, </del> <ins> Mon:      source.CephMonitors, Image:    source.RBDImage, Pool:     pool, Id:       id, Keyring:  keyring, Secret:   secret, </ins> fsType:   source.FSType, <del> readOnly: source.ReadOnly, </del> <ins> ReadOnly: source.ReadOnly, </ins> manager:  manager, mounter:  mounter, plugin:   plugin,

} type rbd struct { <del> volName  string podUID   types.UID mon      []string pool     string id       string image    string keyring  string secret   string </del> <ins> volName string podUID  types.UID // capitalized so they can be exported in persistRBD() Mon      []string Pool     string Id       string Image    string Keyring  string Secret   string </ins> fsType   string <del> readOnly bool </del> <ins> ReadOnly bool </ins> plugin   *rbdPlugin mounter  mount.Interface // Utility interface that provides API calls to the provider to attach/detach disks.

// make mountpoint rw/ro work as expected //FIXME revisit pkg/util/mount and ensure rw/ro is implemented as expected mode := "rw" <del> if rbd.readOnly { </del> <ins> if rbd.ReadOnly { </ins> mode = "ro" } rbd.plugin.execCommand("mount", []string{"-o", "remount," + mode, globalPDPath, dir})

package rbd import ( <ins> "encoding/json" </ins> "errors" "fmt" "math/rand"

"time" "github.com/GoogleCloudPlatform/kubernetes/pkg/util/mount" <ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/util/node" </ins> "github.com/GoogleCloudPlatform/kubernetes/pkg/volume" "github.com/golang/glog" )

type RBDUtil struct{} func (util *RBDUtil) MakeGlobalPDName(rbd rbd) string { <del> return makePDNameInternal(rbd.plugin.host, rbd.pool, rbd.image) </del> <ins> return makePDNameInternal(rbd.plugin.host, rbd.Pool, rbd.Image) } func (util *RBDUtil) rbdLock(rbd rbd, lock bool) error { var err error var output, locker string var cmd []byte var secret_opt []string if rbd.Secret != "" { secret_opt = []string{"--key=" + rbd.Secret} } else { secret_opt = []string{"-k", rbd.Keyring} } // construct lock id using host name and a magic prefix lock_id := "kubelet_lock_magic_" + node.GetHostname("") l := len(rbd.Mon) // avoid mount storm, pick a host randomly start := rand.Int() % l // iterate all hosts until mount succeeds. for i := start; i < start+l; i++ { mon := rbd.Mon[i%l] // cmd "rbd lock list" serves two purposes: // for fencing, check if lock already held for this host // this edge case happens if host crashes in the middle of acquiring lock and mounting rbd // for defencing, get the locker name, something like "client.1234" cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "list", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) output = string(cmd) if err != nil { continue } if lock { // check if lock is already held for this host by matching lock_id and rbd lock id if strings.Contains(output, lock_id) { // this host already holds the lock, exit glog.V(1).Infof("rbd: lock already held for %s", lock_id) return nil } // hold a lock: rbd lock add cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "add", rbd.Image, lock_id, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) } else { // defencing, find locker name ind := strings.LastIndex(output, lock_id) - 1 for i := ind; i >= 0; i-- { if output[i] == 'n' { locker = output[(i + 1):ind] break } } // remove a lock: rbd lock remove cmd, err = rbd.plugin.execCommand("rbd", append([]string{"lock", "remove", rbd.Image, lock_id, locker, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon}, secret_opt...)) } if err == nil { //lock is acquired break } } return err } func (util *RBDUtil) persistRBD(rbd rbd, mnt string) error { file := path.Join(mnt, "rbd.json") fp, err := os.Create(file) if err != nil { return fmt.Errorf("rbd: create err %s/%s", file, err) } defer fp.Close() encoder := json.NewEncoder(fp) if err = encoder.Encode(rbd); err != nil { return fmt.Errorf("rbd: encode err: %v.", err) } return nil } func (util *RBDUtil) loadRBD(rbd *rbd, mnt string) error { file := path.Join(mnt, "rbd.json") fp, err := os.Open(file) if err != nil { return fmt.Errorf("rbd: open err %s/%s", file, err) } defer fp.Close() decoder := json.NewDecoder(fp) if err = decoder.Decode(rbd); err != nil { return fmt.Errorf("rbd: decode err: %v.", err) } return nil } func (util *RBDUtil) fencing(rbd rbd) error { // no need to fence readOnly if rbd.ReadOnly { return nil } return util.rbdLock(rbd, true) } func (util *RBDUtil) defencing(rbd rbd) error { // no need to fence readOnly if rbd.ReadOnly { return nil } return util.rbdLock(rbd, false) </ins> } func (util *RBDUtil) AttachDisk(rbd rbd) error { var err error <del> devicePath := strings.Join([]string{"/dev/rbd", rbd.pool, rbd.image}, "/") </del> <ins> devicePath := strings.Join([]string{"/dev/rbd", rbd.Pool, rbd.Image}, "/") </ins> exist := waitForPathToExist(devicePath, 1) if !exist { // modprobe

return fmt.Errorf("rbd: failed to modprobe rbd error:%v", err) } // rbd map <del> l := len(rbd.mon) </del> <ins> l := len(rbd.Mon) </ins> // avoid mount storm, pick a host randomly start := rand.Int() % l // iterate all hosts until mount succeeds. for i := start; i < start+l; i++ { <del> mon := rbd.mon[i%l] </del> <ins> mon := rbd.Mon[i%l] </ins> glog.V(1).Infof("rbd: map mon %s", mon) <del> if rbd.secret != "" { </del> <ins> if rbd.Secret != "" { </ins> _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "--key=" + rbd.secret}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "--key=" + rbd.Secret}) </ins> } else { _, err = rbd.plugin.execCommand("rbd", <del> []string{"map", rbd.image, "--pool", rbd.pool, "--id", rbd.id, "-m", mon, "-k", rbd.keyring}) </del> <ins> []string{"map", rbd.Image, "--pool", rbd.Pool, "--id", rbd.Id, "-m", mon, "-k", rbd.Keyring}) </ins> } if err == nil { break

return fmt.Errorf("rbd: failed to mkdir %s, error", globalPDPath) } <ins> // fence off other mappers if err := util.fencing(rbd); err != nil { return fmt.Errorf("rbd: image %s is locked by other nodes", rbd.Image) } // rbd lock remove needs ceph and image config // but kubelet doesn't get them from apiserver during teardown // so persit rbd config so upon disk detach, rbd lock can be removed // since rbd json is persisted in the same local directory that is used as rbd mountpoint later, // the json file remains invisible during rbd mount and thus won't be removed accidentally. util.persistRBD(rbd, globalPDPath) </ins> if err = rbd.mounter.Mount(devicePath, globalPDPath, rbd.fsType, nil); err != nil { err = fmt.Errorf("rbd: failed to mount rbd volume %s [%s] to %s, error %v", devicePath, rbd.fsType, globalPDPath, err) }

if err != nil { return fmt.Errorf("rbd: failed to unmap device %s:Error: %v", device, err) } <ins> // load ceph and image/pool info to remove fencing if err := util.loadRBD(&rbd, mntPath); err == nil { // remove rbd lock util.defencing(rbd) } </ins> glog.Infof("rbd: successfully unmap device %s", device) } return nil