Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Network for all nodes of kubernetes cluster (29 hosts) was turned off for a day. Once the network was brought back, we've scaled down a statefulset that mounts a from local volume which on a root drive and then scaled it back up to previous number of replicas. kubelet on 26 out of 29 hosts proceeded with removing all data from subpaths of the pod. In fact, it was ongoing, such that when we'd create a file on a mounted subpath within the container, the file would be cleaned up within a couple of seconds. The issue would continue until volume-subpath was manually unmounted (with command). Ended up removing 3rd of all of our data (we have 3 drives per host and one was wiped) for the whole cluster. Equivalent definition. Note that host path of local volume has to be on root drive: Kubelet logs (filtered out logs from other pods) during the issue: What you expected to happen: Expected no data loss for peristent local volumes. How to reproduce it (as minimally and precisely as possible): Unfortunately was not able to reproduce. But this bug hit 26 out of 29 nodes simultaneously. Need some way to create a case where a pod becomes orphan and "volumes" folder gets somehow removed (similar to issue ). Then there is a code path that can lead to data loss: We are in orphan pod cleanup and directory is missing (not sure why it's printed twice): No error returned and empty volumes list, so we go into this case which proceeds removing pod directory and along with it even thought they might still be mounted (not sure why): checks if device is the same, but we use root drive for local volume, so the device is the same, so it's not a mount point so ok for removing (this is a bug IMHO). Anything else we need to know?: k8s version upgrades went from 1.9.6 -1.11.1 -1.12.1 -1.12.2 -1.13.0 without restarting the pods. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OVH, baremetal servers OS (e.g. from /etc/os-release):Kernel (e.g. ):Install tools: Using Ansible tasks that downloads kubelet from and all other required tools. Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Thank you for digging into the code. I think there are a number of things that can be done here: why returned no volumes. why base volume was successfully unmounted. Unmounting the volume should have failed because the subpath bind mount would have kept a reference on it. to also look for the volume-subpaths directory to detect bind mounts.
I am working on a fix for 3 and 4
, do you mind to share us the full kubelet log? You can email me jinxu at Thanks!
why podVolumesExist returned no volumes. From the available log, we can see that Unmount volume and Unmount device succeeded. That's why podVolumeExist will return no volume.
/priority critical-urgent
That's unexpected. The subpath bind mount should have prevented the local volume bind mount from successfully unmounting.
Maybe the fact that the local volume mount is itself a bind mount could be the reason
about to why podVolumesExist returned no volumes. I think we do have an issue here podVolumesExist checks pod volume paths and whether they are mounted. Suppose it should stop cleaning up if there are paths still mounted. However, it only checks normal pod volumes /pod-id/volumes, it does not check subPath /pod-id/volume-subpath directories. We should fix this.
I'm adding a check for volume-subpath directories to address 3)
Jing found that the subpath volume directory uses the outer volume spec name, which is wrong. We made a fix for this in 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16, which changed the on-disk directories to use the inner volume spec name. had an that in-place node updates were not supported and you would have to drain your pods first. This explains why the volume cleanup proceeded without waiting for the subpath mount cleanup.
We haven't run a version below 1.9.6 on this node.
I've sent the entire kubelet log to your email
to clarify, 1.9.6 uses the old on-disk directory name, so upgrading to anything at least 1.11.0, 1.10.0, 1.9.7, 1.8.11, 1.7.16 requires pod draining first.
Right, I remember stumbling upon when debugging this. However, the release notes say: Shouldn't it specify exact version range where the OuterVolumeSpecName was used (1.9.4-1.9.6)? Unless I'm misreading release notes. I think it would be useful to clarify that.
ah you're right. The release notes should say from at least versions 1.7.14, 1.8.9, and 1.9.4
(will update it)
regardless, I still intend to add more safety checks to this orphaned pod cleanup function
From the log provided by the customer, I think we can confirm the reason of this issue is what mentioned By adding the safety checks during pod cleanup will help avoid this issue.

Discussion:
"kubelet_pods_windows_test.go", "kubelet_resources_test.go", "kubelet_test.go", <ins> "kubelet_volumes_linux_test.go", </ins> "kubelet_volumes_test.go", "oom_watcher_test.go", "pod_container_deletor_test.go",

const ( DefaultKubeletPodsDirName                = "pods" DefaultKubeletVolumesDirName             = "volumes" <ins> DefaultKubeletVolumeSubpathsDirName      = "volume-subpaths" </ins> DefaultKubeletVolumeDevicesDirName       = "volumeDevices" DefaultKubeletPluginsDirName             = "plugins" DefaultKubeletPluginsRegistrationDirName = "plugins_registry"

cadvisorapiv1 "github.com/google/cadvisor/info/v1" "k8s.io/klog" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config"

return filepath.Join(kl.getPodsDir(), string(podUID)) } <ins> // getPodVolumesSubpathsDir returns the full path to the per-pod subpaths directory under // which subpath volumes are created for the specified pod.  This directory may not // exist if the pod does not exist or subpaths are not specified. func (kl *Kubelet) getPodVolumeSubpathsDir(podUID types.UID) string { return filepath.Join(kl.getPodDir(podUID), config.DefaultKubeletVolumeSubpathsDirName) } </ins> // getPodVolumesDir returns the full path to the per-pod data directory under // which volumes are created for the specified pod.  This directory may not // exist if the pod does not exist.

return mountedVolumes, nil } <ins> // podVolumesSubpathsDirExists returns true if the pod volume-subpaths directory for // a given pod exists func (kl *Kubelet) podVolumeSubpathsDirExists(podUID types.UID) (bool, error) { podVolDir := kl.getPodVolumeSubpathsDir(podUID) if pathExists, pathErr := volumeutil.PathExists(podVolDir); pathErr != nil { return true, fmt.Errorf("Error checking if path %q exists: %v", podVolDir, pathErr) } else if !pathExists { return false, nil } return true, nil } </ins> // GetVersionInfo returns information about the version of cAdvisor in use. func (kl *Kubelet) GetVersionInfo() (*cadvisorapiv1.VersionInfo, error) { return kl.cadvisor.VersionInfo()

import ( "fmt" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/types" utilerrors "k8s.io/apimachinery/pkg/util/errors" "k8s.io/apimachinery/pkg/util/sets"

} // If volumes have not been unmounted/detached, do not delete directory. // Doing so may result in corruption of data. <ins> // TODO: getMountedVolumePathListFromDisk() call may be redundant with // kl.getPodVolumePathListFromDisk(). Can this be cleaned up? </ins> if podVolumesExist := kl.podVolumesExist(uid); podVolumesExist { klog.V(3).Infof("Orphaned pod %q found, but volumes are not cleaned up", uid) continue

orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but volume paths are still present on disk", uid)) continue } <ins> // If there are any volume-subpaths, do not cleanup directories volumeSubpathExists, err := kl.podVolumeSubpathsDirExists(uid) if err != nil { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but error %v occurred during reading of volume-subpaths dir from disk", uid, err)) continue } if volumeSubpathExists { orphanVolumeErrors = append(orphanVolumeErrors, fmt.Errorf("Orphaned pod %q found, but volume subpaths are still present on disk", uid)) continue } </ins> klog.V(3).Infof("Orphaned pod %q found, removing", uid) if err := removeall.RemoveAllOneFilesystem(kl.mounter, kl.getPodDir(uid)); err != nil { klog.Errorf("Failed to remove orphaned pod %q dir; err: %v", uid, err)

<ins> // +build linux /* Copyright 2018 The Kubernetes Authors. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ package kubelet import ( "fmt" "io/ioutil" "os" "path/filepath" "testing" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" _ "k8s.io/kubernetes/pkg/apis/core/install" ) func validateDirExists(dir string) error { _, err := ioutil.ReadDir(dir) if err != nil { return err } return nil } func validateDirNotExists(dir string) error { _, err := ioutil.ReadDir(dir) if os.IsNotExist(err) { return nil } if err != nil { return err } return fmt.Errorf("dir %q still exists", dir) } func TestCleanupOrphanedPodDirs(t *testing.T) { testCases := map[string]struct { pods         []*v1.Pod prepareFunc  func(kubelet *Kubelet) error validateFunc func(kubelet *Kubelet) error expectErr    bool }{ "nothing-to-do": {}, "pods-dir-not-found": { prepareFunc: func(kubelet *Kubelet) error { return os.Remove(kubelet.getPodsDir()) }, expectErr: true, }, "pod-doesnot-exist-novolume": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return os.MkdirAll(filepath.Join(podDir, "not/a/volume"), 0750) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirNotExists(filepath.Join(podDir, "not")) }, }, "pod-exists-with-volume": { pods: []*v1.Pod{ { ObjectMeta: metav1.ObjectMeta{ Name: "pod1", UID:  "pod1uid", }, }, }, prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return os.MkdirAll(filepath.Join(podDir, "volumes/plugin/name"), 0750) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volumes/plugin/name")) }, }, "pod-doesnot-exist-with-volume": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return os.MkdirAll(filepath.Join(podDir, "volumes/plugin/name"), 0750) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volumes/plugin/name")) }, }, "pod-doesnot-exist-with-subpath": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return os.MkdirAll(filepath.Join(podDir, "volume-subpaths/volume/container/index"), 0750) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volume-subpaths/volume/container/index")) }, }, "pod-doesnot-exist-with-subpath-top": { prepareFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return os.MkdirAll(filepath.Join(podDir, "volume-subpaths"), 0750) }, validateFunc: func(kubelet *Kubelet) error { podDir := kubelet.getPodDir("pod1uid") return validateDirExists(filepath.Join(podDir, "volume-subpaths")) }, }, // TODO: test volume in volume-manager } for name, tc := range testCases { t.Run(name, func(t *testing.T) { testKubelet := newTestKubelet(t, false /* controllerAttachDetachEnabled */) defer testKubelet.Cleanup() kubelet := testKubelet.kubelet if tc.prepareFunc != nil { if err := tc.prepareFunc(kubelet); err != nil { t.Fatalf("%s failed preparation: %v", name, err) } } err := kubelet.cleanupOrphanedPodDirs(tc.pods, nil) if tc.expectErr && err == nil { t.Errorf("%s failed: expected error, got success", name) } if !tc.expectErr && err != nil { t.Errorf("%s failed: got error %v", name, err) } if tc.validateFunc != nil { if err := tc.validateFunc(kubelet); err != nil { t.Errorf("%s failed validation: %v", name, err) } } }) } } </ins>

fsckErrorsUncorrected = 4 // place for subpath mounts <ins> // TODO: pass in directory using kubelet_getters instead </ins> containerSubPathDirectoryName = "volume-subpaths" // syscall.Openat flags used to traverse directories not following symlinks nofollowFlags = unix.O_RDONLY | unix.O_NOFOLLOW