Change:
While debugging a node-e2e failure on Ubuntu image, I noticed that the kublet log is full of the following error messages from: The above failure results in tons of EmptyDir volumes (including both secret and confgmap) are not properly deleted, thus test failures. cc/
CC
could you share the complete kubelet logs please
you can access the kubelet log from previous run at: They have the same issue.
for the most recent test run see also
Is there a link not behind a login?
Here is one of the errors: Saad and I debugged. Our hypothesis is that: The Rename() method for EmptyDir is first creating a directory with a new name and then tries to rename the old directory to the new name. That os.Rename() is failing on Ubuntu because the directory already exists, which succeeds on other images. To verify the Ubuntu issue, we can write a small test program using os.Rename() directly on the Ubuntu image. Then the volume reconstruction logic is picking up the new directory as a new emptydir volume for the pod, which triggers another deletion round.
Some things we should improve regardless of the root cause of this: volume reconstruction logic should ignore any folders (to break the explosion of directories/volumes in cases like this). the the default method for : Understand why it does a create directory before rename. Eliminate the create directory if possible. Otherwise make sure that the created directory is deleted if subsequent rename fails. Understand why a rename is required at all before delete. Eliminate it if possible. Otherwise make sure that if delete fails, that the rename is undone.
BTW, in Go 1.8, it looks like they've changed the behavior of os.Rename: So I think we should move towards eliminating the create directory.
Does this mean the problem could be caused by a difference in the use of golang 1.7 vs. golang 1.8 for k8s binary builds?
Yes that's possible. From the development guide, go1.8 has not been verified for building k8s.
related
Is the test binary which generates the failures reported here is built with golang 1.8? I couldn't find those information through the existing logs.
From what I understand, the purpose of the rename is so that the system knows quickly that the volume has been deleted. This is important for volumes with large content that can take a long time to delete. So I think the right set of changes that are needed are: The volume reconstruction logic should ignore directories with ".deleting~". I can make this change. The rename needs to be changed to not create a directory first. It looks like is handling this.
On Thu, Mar 23, 2017 at 01:27:30PM -0700, Dawn Chen wrote: Yes, it is. Should we build with golang-1.7 instead?
Confirmed that when building with go1.7 instead of go1.8, the node tests succeed.
Great! Thanks for the updates. We can continue carrying on our integration work. And the issue has nothing to do with Ubuntu image itself. I am leaving this open to fix the error-prone logic in EmptyDir's teardown logic, and resolving the golang 1.8 compatibility issue.
looks like a possible dup of
I've encountered this issue about a month ago and now I can confirm that this issue is caused by compiling with Go 1.8, which forbids renaming a directory into an existing directory even if the destinition is empty. So the current logic that uses call does not work properly under Go 1.8. The correct way to solve it is to change the behavior in so that it does not rely on the call to which creates a new empty directory after a successful call.
I discussed with Saad. We think that after the volume manager redesign, we don't need to do a rename at all. The volume operation executor can handle duplicate requests for volumes with the same name. I will work on testing a fix for this.

Discussion:
} func (ed *emptyDir) teardownDefault(dir string) error { <del> tmpDir, err := volume.RenameDirectory(dir, ed.volName+".deleting~") if err != nil { return err } err = os.RemoveAll(tmpDir) </del> <ins> // Renaming the directory is not required anymore because the operation executor // now handles duplicate operations on the same volume err := os.RemoveAll(dir) </ins> if err != nil { return err }

package volume import ( <del> "io" "io/ioutil" "os" filepath "path/filepath" "runtime" </del> "time" <del> "github.com/golang/glog" </del> "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/types"

func (err deletedVolumeInUseError) Error() string { return string(err) } <del> func RenameDirectory(oldPath, newName string) (string, error) { newPath, err := ioutil.TempDir(filepath.Dir(oldPath), newName) if err != nil { return "", err } // os.Rename call fails on windows (https://github.com/golang/go/issues/14527) // Replacing with copyFolder to the newPath and deleting the oldPath directory if runtime.GOOS == "windows" { err = copyFolder(oldPath, newPath) if err != nil { glog.Errorf("Error copying folder from: %s to: %s with error: %v", oldPath, newPath, err) return "", err } os.RemoveAll(oldPath) return newPath, nil } err = os.Rename(oldPath, newPath) if err != nil { return "", err } return newPath, nil } func copyFolder(source string, dest string) (err error) { fi, err := os.Lstat(source) if err != nil { glog.Errorf("Error getting stats for %s. %v", source, err) return err } err = os.MkdirAll(dest, fi.Mode()) if err != nil { glog.Errorf("Unable to create %s directory %v", dest, err) } directory, _ := os.Open(source) defer directory.Close() objects, err := directory.Readdir(-1) for _, obj := range objects { if obj.Mode()&os.ModeSymlink != 0 { continue } sourceFilePointer := source + "" + obj.Name() destinationFilePointer := dest + "" + obj.Name() if obj.IsDir() { err = copyFolder(sourceFilePointer, destinationFilePointer) if err != nil { return err } } else { err = copyFile(sourceFilePointer, destinationFilePointer) if err != nil { return err } } } return } func copyFile(source string, dest string) (err error) { sourceFile, err := os.Open(source) if err != nil { return err } defer sourceFile.Close() destFile, err := os.Create(dest) if err != nil { return err } defer destFile.Close() _, err = io.Copy(destFile, sourceFile) if err == nil { sourceInfo, err := os.Stat(source) if err != nil { err = os.Chmod(dest, sourceInfo.Mode()) } } return } </del>