Code change:
When mounting several shares over NFS to container and when one of the directories is deleted from NFS server, something in the node is deleting content from other shares. I suspect that something goes wrong with "cleanupOrphanedVolumes"? not sure tough. Use case: php-app is running in container, uses php-sessions for remembering login. php sessions are stored in file and shared through nfs-share between containers. also app code is shared and several other directories to store different data (data dirs are mapped under code, code is in separate share so generated document files don't get to git tree), all is managed by separate integration server (in container), that uses same shares to prepare the code. This happened first time in GCE with Cluster API version 0.17.something, now it acts same on 1.0.1 and 0.18.2. Problem is repeatable: First of all you need to install package "nfs-common" to all cluster nodes, otherwise you can't map NFS shares at all: now you need nfs server. I created small instance for that with /etc/export stating: also I made directories under /export/test/: next you need to start it up (replace nfs server ip with your own): now start up the pod: verify, that the directories are mapped: now the funny part: remove directory from nfs server the zubuntu pod shows: nothing happens on nfs server yet, but just launch the stop command: and now nfs server shows: you see, that one file was left, but not for long. this is because root in client does not have root access in nfs-file system, but just after giving rights, the file is gone: now you can test "black hole" by creating new files. they dissaperar from the server in few moments. the dissapearing continues until cluster node is restarted.
so the problem looks to me is that containers delete files though they are not told to do. I am not quite convinced is the culprit here, since the directories are still alive. what the first version you believe this problem started to show up? Would it be possible for you to take a tcpdump on the nfs server so we can look at the nfs traffic? Thanks!
I started to play with the Google Container Engine about in middle of June and the version of the cluster was 0.17. something. I don’t have that cluster anymore. So I don’t know what happened or did it worked before. About tcpdump. What ports and what traffic do you think you need for that? Also in what timeframe? Maybe you can even give me command-line for that? Jüri.
I am setting up an environment to simulate your tests, stay tuned
yes, I can reproduce your issue. And I was wrong about . You are more likely to be right on that suspect. Blame me if I point you to the wrong suspect. Here in this line , doesn't return any error if a volume fails to be torn down, which is what happening here: the first volume is not able to be torn down, yet the error message is suppressed, so later on thinks volumes are unmounted and it is safe to remove the directory
Something is fishy. Volume cleanup should be happening against the volume dirs, which is totally separate from pod dirs. Pod teardown should un-bind-mount the NFS from th epod dir, so pod cleanup can happen, even if the volume can't umount...
I didn't touch much of the volume code, but think the directory structure is . Removing pod directories would remove the volumes as well. We can delay the pod directory cleanup until all volumes in it are properly unmounted. Another user reported a similar case in
in this particular case, the mountpoint cannot be properly unmounted (because it was deleted), and I believe the current NFS mount is , so it won't be easy to unmount even after delay. unmount error might be a good heuristic. In this case, NFS stale handle error is probably good sign nothing is good for wait for.
Ahh yes, it was only GCE PD that was doing something tricker here , Yu-Ju Hong wrote:
I see. Thanks for the explanation. I didn't read the original post carefully. If we let propagate the error upstream, kubelet may always skip pod directory cleanup because of one failed teardown. That seems undesirable. I agree that the NFS stale handle error alone should be enough for us to skip waiting. In short, should distinguish the NFS error and report a successful teardown in this case. should check if there are any uncleaned volume directories, and avoid removing the specific pod.
cool, thanks. When a patch is available, I'll give it a test on other volumes too.
This was completed by in PR can you take a crack at this?
ok, glad to do it.
Thanks!

Related discussion:
text: for _, volumeKindDir := range volumeKindDirs { volumeKind := volumeKindDir.Name() volumeKindPath := path.Join(podVolDir, volumeKind) <del> volumeNameDirs, err := ioutil.ReadDir(volumeKindPath) </del> <ins> // ioutil.ReadDir exits without returning any healthy dir when encountering the first lstat error // but skipping dirs means no cleanup for healthy volumes. switching to a no-exit api solves this problem volumeNameDirs, volumeNameDirsStat, err := util.ReadDirNoExit(volumeKindPath) </ins> if err != nil { return []*volumeTuple{}, fmt.Errorf("could not read directory %s: %v", volumeKindPath, err) } <del> for _, volumeNameDir := range volumeNameDirs { volumes = append(volumes, &volumeTuple{Kind: volumeKind, Name: volumeNameDir.Name()}) </del> <ins> for i, volumeNameDir := range volumeNameDirs { if volumeNameDir != nil { volumes = append(volumes, &volumeTuple{Kind: volumeKind, Name: volumeNameDir.Name()}) } else { lerr := volumeNameDirsStat[i] glog.Errorf("Could not read directory %s: %v", podVolDir, lerr) } </ins> } } return volumes, nil