Change:
Fix status: [x] fixed in master in [x] fixed in release-1.10 branch in , will be in 1.10.0 [x] fixed in release-1.9 branch in , will be in 1.9.7 [x] fixed in release-1.8 branch in , will be in 1.8.11 [x] fixed in release-1.7 branch in , will be in 1.7.16 In SubPath fixes, PR adds bind mount for subpaths. It uses as the path name. The volumename in this path is the container's mount volume name which is also called OuterVolumeSpecName in . This becomes an issue for volume reconstruction when kubelet restarts. Volume scans the volume directories and recover the InnerVolumeSpecName (which is used to construct pod volume paths). Volume reconstruction has not enough information to recover the OuterVolumeSpecName and figure out which volume it maps to. In case kubelet needs to clean up mounts (it happens when volume spec cannot be reconstructed), it will fail to clean up the subpaths because of lacking the OuterVolumeSpecName. We think it is better to use InnerVolumeSpecName for volume subPath instead of OuterVolumeSpecName so that volume clean up can work. The issue we might have is that some users might already deploy a cluster with subpath might already . If the users perform an in-place upgrade, since new version will use InnerVolumeSpecName which might have conflicts with some other older version's OuterVolumeSpecName among different volumes. This change requires clear documentation and communication with the users.
cc
This impacts reconstruction of PVCs, where the inner and outer names are different.
/assign
While testing the e2es against a cluster without the fix, I noticed that subsequent pods using volumes failed to start. My theory is that the unmount volume failed because the subpath was still mounted, and that blocked the volume manager from processing other pods. But I thought that volume manager is supposed to run the unmount volume operation in a separate thread per volume, so this shouldn't happen. Need to inspect the code more carefully to confirm the theory. could the volume reconstruction refactor have a unmount volume call in the main volume manager thread?
runs in reconciler's goroutine - so if that function is blocked, it will not be able to process any more volumes. Was the problem that failed to unmount volume but got hung/stuck ?
I haven't confirmed for sure, but my guess is that unmount volume failed because the subpath was still mounted
moving into the milestone for discussion /milestone v1.10
/priority critical-urgent /status in-progress
/status approved-for-milestone per discussion in sig-release meeting today
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Re-opening to track fixes in release branches
keeping in the v1.10 milestone until merges
since this is now fixed in 1.10, I'm taking it out of the milestone so that the release team doesn't keep checking it.
Can this issue be closed now?

Discussion:
// Whether the volume permission is set to read-only or not // This value is passed from volume.spec ReadOnly bool <ins> // Inner volume spec name, which is the PV name if used, otherwise // it is the same as the outer volume spec name. InnerVolumeSpecName string </ins> } type VolumeMap map[string]VolumeInfo

hostPath, cleanupAction, err = mounter.PrepareSafeSubpath(mountutil.Subpath{ VolumeMountIndex: i, Path:             hostPath, <del> VolumeName:       mount.Name, </del> <ins> VolumeName:       vol.InnerVolumeSpecName, </ins> VolumePath:       volumePath, PodDir:           podDir, ContainerName:    container.Name,

podVolumes := make(container.VolumeMap) for _, mountedVolume := range vm.actualStateOfWorld.GetMountedVolumesForPod(podName) { podVolumes[mountedVolume.OuterVolumeSpecName] = container.VolumeInfo{ <del> Mounter:           mountedVolume.Mounter, BlockVolumeMapper: mountedVolume.BlockVolumeMapper, ReadOnly:          mountedVolume.VolumeSpec.ReadOnly, </del> <ins> Mounter:             mountedVolume.Mounter, BlockVolumeMapper:   mountedVolume.BlockVolumeMapper, ReadOnly:            mountedVolume.VolumeSpec.ReadOnly, InnerVolumeSpecName: mountedVolume.InnerVolumeSpecName, </ins> } } return podVolumes

// This implementation is shared between Linux and NsEnterMounter func doCleanSubPaths(mounter Interface, podDir string, volumeName string) error { <del> glog.V(4).Infof("Cleaning up subpath mounts for %s", podDir) </del> // scan /var/lib/kubelet/pods/<uid>/volume-subpaths/<volume>/* subPathDir := filepath.Join(podDir, containerSubPathDirectoryName, volumeName) <ins> glog.V(4).Infof("Cleaning up subpath mounts for %s", subPathDir) </ins> containerDirs, err := ioutil.ReadDir(subPathDir) if err != nil { if os.IsNotExist(err) {

// Remove all bind-mounts for subPaths podDir := path.Join(podsDir, string(volumeToUnmount.PodUID)) <del> if err := mounter.CleanSubPaths(podDir, volumeToUnmount.OuterVolumeSpecName); err != nil { </del> <ins> if err := mounter.CleanSubPaths(podDir, volumeToUnmount.InnerVolumeSpecName); err != nil { </ins> return volumeToUnmount.GenerateError("error cleaning subPath mounts", err) }

"k8s.io/api/core/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" <ins> apierrs "k8s.io/apimachinery/pkg/api/errors" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" utilerrors "k8s.io/apimachinery/pkg/util/errors"

"hostPath":         initHostpath, "hostPathSymlink":  initHostpathSymlink, "emptyDir":         initEmptydir, <del> "gcePD":            initGCEPD, </del> <ins> "gcePDPVC":         initGCEPD, </ins> "gcePDPartitioned": initGCEPDPartition, "nfs":              initNFS, "nfsPVC":           initNFSPVC,

testPodContainerRestart(f, pod, filePathInVolume, filePathInSubpath) }) <ins> It("should unmount if pod is gracefully deleted while kubelet is down [Disruptive][Slow]", func() { testSubpathReconstruction(f, pod, false) }) It("should unmount if pod is force deleted while kubelet is down [Disruptive][Slow]", func() { if curVolType == "hostPath" || curVolType == "hostPathSymlink" { framework.Skipf("%s volume type does not support reconstruction, skipping", curVolType) } testSubpathReconstruction(f, pod, true) }) </ins> }) }

Expect(strings.TrimSpace(out)).To(BeEquivalentTo("test-after")) } <ins> func testSubpathReconstruction(f *framework.Framework, pod *v1.Pod, forceDelete bool) { // This is mostly copied from TestVolumeUnmountsFromDeletedPodWithForceOption() // Change to busybox pod.Spec.Containers[0].Image = "busybox" pod.Spec.Containers[0].Command = []string{"/bin/sh", "-ec", "sleep 100000"} pod.Spec.Containers[1].Image = "busybox" pod.Spec.Containers[1].Command = []string{"/bin/sh", "-ec", "sleep 100000"} By(fmt.Sprintf("Creating pod %s", pod.Name)) pod, err := f.ClientSet.CoreV1().Pods(f.Namespace.Name).Create(pod) Expect(err).ToNot(HaveOccurred()) err = framework.WaitTimeoutForPodRunningInNamespace(f.ClientSet, pod.Name, pod.Namespace, time.Minute) Expect(err).ToNot(HaveOccurred()) pod, err = f.ClientSet.CoreV1().Pods(f.Namespace.Name).Get(pod.Name, metav1.GetOptions{}) Expect(err).ToNot(HaveOccurred()) nodeIP, err := framework.GetHostExternalAddress(f.ClientSet, pod) Expect(err).NotTo(HaveOccurred()) nodeIP = nodeIP + ":22" By("Expecting the volume mount to be found.") result, err := framework.SSH(fmt.Sprintf("mount | grep %s | grep -v volume-subpaths", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Code).To(BeZero(), fmt.Sprintf("Expected grep exit code of 0, got %d", result.Code)) By("Expecting the subpath volume mount to be found.") result, err = framework.SSH(fmt.Sprintf("cat /proc/self/mountinfo | grep volume-subpaths | grep %s", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Code).To(BeZero(), fmt.Sprintf("Expected grep exit code of 0, got %d", result.Code)) By("Stopping the kubelet.") utils.KubeletCommand(utils.KStop, f.ClientSet, pod) defer func() { if err != nil { utils.KubeletCommand(utils.KStart, f.ClientSet, pod) } }() By(fmt.Sprintf("Deleting Pod %q", pod.Name)) if forceDelete { err = f.ClientSet.CoreV1().Pods(pod.Namespace).Delete(pod.Name, metav1.NewDeleteOptions(0)) } else { err = f.ClientSet.CoreV1().Pods(pod.Namespace).Delete(pod.Name, &metav1.DeleteOptions{}) } Expect(err).NotTo(HaveOccurred()) By("Starting the kubelet and waiting for pod to delete.") utils.KubeletCommand(utils.KStart, f.ClientSet, pod) err = f.WaitForPodTerminated(pod.Name, "") if !apierrs.IsNotFound(err) && err != nil { Expect(err).NotTo(HaveOccurred(), "Expected pod to terminate.") } if forceDelete { // With forceDelete, since pods are immediately deleted from API server, there is no way to be sure when volumes are torn down // so wait some time to finish time.Sleep(30 * time.Second) } By("Expecting the volume mount not to be found.") result, err = framework.SSH(fmt.Sprintf("mount | grep %s | grep -v volume-subpaths", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Stdout).To(BeEmpty(), "Expected grep stdout to be empty (i.e. no mount found).") framework.Logf("Volume unmounted on node %s", pod.Spec.NodeName) By("Expecting the subpath volume mount not to be found.") result, err = framework.SSH(fmt.Sprintf("cat /proc/self/mountinfo | grep volume-subpaths | grep %s", pod.UID), nodeIP, framework.TestContext.Provider) framework.LogSSHResult(result) Expect(err).NotTo(HaveOccurred(), "Encountered SSH error.") Expect(result.Stdout).To(BeEmpty(), "Expected grep stdout to be empty (i.e. no subpath mount found).") framework.Logf("Subpath volume unmounted on node %s", pod.Spec.NodeName) } </ins> func podContainerExec(pod *v1.Pod, containerIndex int, bashExec string) (string, error) { return framework.RunKubectl("exec", fmt.Sprintf("--namespace=%s", pod.Namespace), pod.Name, "--container", pod.Spec.Containers[containerIndex].Name, "--", "/bin/sh", "-c", bashExec) }