Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: There is a node in my cluster whose disk is full on root partition, I cleared the disk and then rebooted the node to recover it. After the reboot, the kube-proxy on that node got stuck in "ContainerCreating" status and never changes, and all other pods can't run properly since the kubeproxy is not runing. What you expected to happen: Kubeproxy on that node can run normally after the disk clean and reboot How to reproduce it (as minimally and precisely as possible): the disk full on root partition in my env, leave roughly 100k space on root partition would make this issue more reproduciblethis issue is kind of tricky, to reproduce it, one need to populate the disk repeatedly since some disk space would be cleared because of pod evictionthe kube-proxy got deleted and created repeatedly because of pod eviction and daemon-set mechanism, it would stuck in ContainerCreating status if the disk space is not enough for creating volume contents of kube-proxy configMap volume, the first pod error event is like: MountVolume.Setup failed for volume "kube-proxy" space left on device After it got stuck, the following pod error event woud repeat over and over, kube-proxy is trying to mount configMap volume but always fail MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory the disk and reboot the reboot is neccesary to reproduce the issue, without the reboot the pod eviction would kick in to delete kube-proxy pod the kube-proxy woud stuck in ContainerCreating Status forever, restart kubelet has no effects on that, kube-proxy pod need to be mannully deleted to get recover kube-proxy pod can't mount configMap volume, the following error keeps repeating MountVolume.Setup failed for volume "kube-proxy": stat /var/lib/kubelet/pods/{podid} no such file or directory Anything else we need to know?: After digging into this issue for a quite long time, i think the reason for this issue is: The configMap volume uses emptyDir internally to create the volume directory, after the directory is created, configMap volume failed to populate it with configMap content because the disk is full, then configMap volume calls emptyDir.tearDown() to clear the temp directory. The problem is that emptyDir uses a "ready" file under /var/lib/kubelet/pods/{podId}/plugin/.. to determine whether the directory is created or not, if the ready file is present, then it wouldn't create the directory. The tearDown method only remove the directory, it doen't remove the ready file, so after the first out of disk error, the diretory creation always get skipped, hence no such file or directory error repeat over and over. Environment: Kubernetes version (use ): v.1.11.5 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): cent os 7 Kernel (e.g. ): Install tools: kubeadm Others:
/sig storage
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
If it's any help, I just encountered this issue in kubelet v1.13.2 under Debian Stretch 9.5, after a restart. I was not out of disk space. Deleting the ready files from /var/lib/kubelet/pods/{podId}*/ helped.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-testing, kubernetes/test-infra and/or . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen hit this just now with the same conditions as originally described, but without a reboot.
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen </details
/reopen
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Thanks ! this helped me get my cluster running after all pods were evicted and clearing space alone didn't fix the issue.

Discussion:
// storage medium is the default, then the volume is ready.  If the // medium is memory, and a mountpoint is present, then the volume is // ready. <del> if volumeutil.IsReady(ed.getMetaDir()) { </del> <ins> readyDir := ed.getMetaDir() if volumeutil.IsReady(readyDir) { </ins> if ed.medium == v1.StorageMediumMemory && !notMnt { return nil } else if ed.medium == v1.StorageMediumDefault { <del> return nil </del> <ins> // Further check dir exists if _, err := os.Stat(dir); err == nil { return nil } // This situation should not happen unless user manually delete volume dir. // In this case, delete ready file and print a warning for it. klog.Warningf("volume ready file dir %s exist, but volume dir %s does not. Remove ready dir", readyDir, dir) if err := os.RemoveAll(readyDir); err != nil && !os.IsNotExist(err) { klog.Warningf("failed to remove ready dir [%s]: %v", readyDir, err) } </ins> } }

// TearDownAt simply discards everything in the directory. func (ed *emptyDir) TearDownAt(dir string) error { <ins> // First remove ready dir which created in SetUp func readyDir := ed.getMetaDir() if removeErr := os.RemoveAll(readyDir); removeErr != nil && !os.IsNotExist(removeErr) { return fmt.Errorf("failed to remove ready dir [%s]: %v", readyDir, removeErr) } </ins> if pathExists, pathErr := mount.PathExists(dir); pathErr != nil { return fmt.Errorf("Error checking if path exists: %v", pathErr) } else if !pathExists {

"path/filepath" "testing" <del> "k8s.io/api/core/v1" </del> <ins> v1 "k8s.io/api/core/v1" </ins> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

"k8s.io/kubernetes/pkg/volume" volumetest "k8s.io/kubernetes/pkg/volume/testing" "k8s.io/kubernetes/pkg/volume/util" <ins> volumeutil "k8s.io/kubernetes/pkg/volume/util" </ins> "k8s.io/mount-utils" )

func TestPluginEmptyRootContext(t *testing.T) { doTestPlugin(t, pluginTestConfig{ <ins> volumeDirExists:        true, readyDirExists:         true, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        false, readyDirExists:         false, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        true, readyDirExists:         false, medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0}) doTestPlugin(t, pluginTestConfig{ volumeDirExists:        false, readyDirExists:         true, </ins> medium:                 v1.StorageMediumDefault, expectedSetupMounts:    0, expectedTeardownMounts: 0})

} type pluginTestConfig struct { <del> medium                        v1.StorageMedium idempotent                    bool </del> <ins> medium v1.StorageMedium //volumeDirExists indicates whether volumeDir already/still exists before volume setup/teardown volumeDirExists bool //readyDirExists indicates whether readyDir already/still exists before volume setup/teardown readyDirExists                bool </ins> expectedSetupMounts           int shouldBeMountedBeforeTeardown bool expectedTeardownMounts        int

} ) <del> if config.idempotent { </del> <ins> if config.readyDirExists { </ins> physicalMounter.MountPoints = []mount.MountPoint{ { Path: volumePath,

if volPath != volumePath { t.Errorf("Got unexpected path: %s", volPath) } <del> if err := mounter.SetUp(volume.MounterArgs{}); err != nil { t.Errorf("Expected success, got: %v", err) </del> <ins> if config.volumeDirExists { if err := os.MkdirAll(volPath, perm); err != nil { t.Errorf("fail to create path: %s", volPath) } </ins> } // Stat the directory and check the permission bits <del> fileinfo, err := os.Stat(volPath) if !config.idempotent { if err != nil { if os.IsNotExist(err) { t.Errorf("SetUp() failed, volume path not created: %s", volPath) } else { t.Errorf("SetUp() failed: %v", err) } } if e, a := perm, fileinfo.Mode().Perm(); e != a { t.Errorf("Unexpected file mode for %v: expected: %v, got: %v", volPath, e, a) } } else if err == nil { // If this test is for idempotency and we were able // to stat the volume path, it's an error. t.Errorf("Volume directory was created unexpectedly") } </del> <ins> testSetUp(mounter, metadataDir, volPath) </ins> log := physicalMounter.GetLog() // Check the number of mounts performed during setup

t.Errorf("Got a nil Unmounter") } <del> // Tear down the volume if err := unmounter.TearDown(); err != nil { t.Errorf("Expected success, got: %v", err) </del> <ins> if !config.readyDirExists { if err := os.RemoveAll(metadataDir); err != nil && !os.IsNotExist(err) { t.Errorf("failed to remove ready dir [%s]: %v", metadataDir, err) } </ins> } <del> if _, err := os.Stat(volPath); err == nil { t.Errorf("TearDown() failed, volume path still exists: %s", volPath) } else if !os.IsNotExist(err) { t.Errorf("TearDown() failed: %v", err) </del> <ins> if !config.volumeDirExists { if err := os.RemoveAll(volPath); err != nil && !os.IsNotExist(err) { t.Errorf("failed to remove ready dir [%s]: %v", metadataDir, err) } } // Tear down the volume if err := testTearDown(unmounter, metadataDir, volPath); err != nil { t.Errorf("Test failed with error %v", err) </ins> } log = physicalMounter.GetLog()

physicalMounter.ResetLog() } <ins> func testSetUp(mounter volume.Mounter, metadataDir, volPath string) error { if err := mounter.SetUp(volume.MounterArgs{}); err != nil { return fmt.Errorf("Expected success, got: %v", err) } // Stat the directory and check the permission bits if !volumeutil.IsReady(metadataDir) { return fmt.Errorf("SetUp() failed, ready file is not created") } fileinfo, err := os.Stat(volPath) if err != nil { if os.IsNotExist(err) { return fmt.Errorf("SetUp() failed, volume path not created: %s", volPath) } return fmt.Errorf("SetUp() failed: %v", err) } if e, a := perm, fileinfo.Mode().Perm(); e != a { return fmt.Errorf("Unexpected file mode for %v: expected: %v, got: %v", volPath, e, a) } return nil } func testTearDown(unmounter volume.Unmounter, metadataDir, volPath string) error { if err := unmounter.TearDown(); err != nil { return err } if volumeutil.IsReady(metadataDir) { return fmt.Errorf("Teardown() failed, ready file still exists") } if _, err := os.Stat(volPath); err == nil { return fmt.Errorf("TearDown() failed, volume path still exists: %s", volPath) } else if !os.IsNotExist(err) { return fmt.Errorf("TearDown() failed: %v", err) } return nil } </ins> func TestPluginBackCompat(t *testing.T) { basePath, err := utiltesting.MkTmpdir("emptydirTest") if err != nil {