Change:
hostPath = filepath.Join(hostPath, mount.SubPath) <del> // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } </del> <ins> if subPathExists, err := util.FileExists(hostPath); err != nil { glog.Errorf("Could not determine if subPath %s exists; will not attempt to change its permissions", hostPath) } else if !subPathExists { // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } </ins> <del> // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err </del> <ins> // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err } </ins> } }

Discussion:
k8s 1.6.3 just landed and broke a lot of our systems. The issue shows as: 2017-05-10 20:07:19. | NAME READY STATUS RESTARTS AGE 2017-05-10 20:07:19. | test-ceph-init-mon-hx3h1 0/1 mkdir /var/lib/kubelet/pods/-35bb-11e7-8ad3- not a directory 0 3m The issue is, we have subPath set to a file in a configmap to map it into the right place in our container. But the new code assumes subPath is a directory and always does a mkdir over it. This causes things to fail. The PR in question is:
cc /assign wongma7
Reiterating the mentions to trigger a notification: <detailsIn response to : cc /assign wongma7 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
Its also related to:
can you ensure your fix is picked to 1.6 branch, too?
yes, I will.
any chance it gets finally committed? our testing gates are all red because of this issue..
I brought it up in the sig-storage meeting today so priority was raised to critical-urgent and sig node was pinged. I guess we could make some more noise, I'll try finding people on slack
Would be great to get it out one way or another. I have heard other people also see this issue.
Status: it's merging to master right now & I just opened a cherry-pick for it into 1.6 I'm not sure what the procedure is for expediting a point release, but I suppose it would be enough if those other people +1 this issue, it may help the folks in charge judge the severity of the bug. and what's the procedure here? The context is that a bug was introduced with 1.6.3 that makes it so that certain pods (that mount a volume's file using the subPath field) fail to start. Thanks!
FYI, closed by mistake, sorry.

We still need to track it 'til it's fixed in 1.6.x
I think an expedited release would be warranted if there are a large number of users who both (1) are affected by this issue, and (2) need something in 1.6.3 so rolling back to 1.6.2 is not good enough. I don't know how common it is for users to rely on this feature, so any signal we can get on that would be helpful.
this feature is used a lot by kolla-kubernetes project. Rolling back is problematic, our gating system works with centos and Ubuntu images, with centos we can go back to 1.6.2 but with Ubuntu, it looks like 1.6.2 binaries have been already removed. We use binaries which are provided by repo and not static files.
Perhaps we should add an e2e test case for subPath.
yeah, i have to agree with and on this one. our projects are very similar, and we're experiencing the same thing on our end w/v1.6.3.
After discussing with I am planning to expedite v1.6.4 to fix this. However, I will wait for the e2e test, which I believe is working on.
Keeping open until v1.6.4 ships with fix.
the e2e test PR is here: . They cover both this issue and the original one I was trying to fix, .
should kubernetes yank the binary packages for kubelet/kubeadm/etc that were released for 1.6.3 (debian/rpm/etc) - until 1.6.4 gets released? Because many of us who are just installing kubeadm/kubelet/etc with 1.6.* have found that things suddenly start break. Right now, its in the release note with a comment like . is not an alpha/beta feature, it's a core feature, and we just can't assume someone "may or may not" use it. Should this be rather more critical than that? 3 is a broken release and nobody should ever install it right?
as a workaround we compiled 1.6.3 kubelet with the one fix. If you are interested, please let me know I will share the link, I have it posted on GitHub.

thank you, we have pinned kubernetes versions for now... I can see that k8s 1.6.4-beta.1 is already being tagged. Great work from the team! Looks like k8s 1.6.4 will supersede 1.6.3 shortly. Thanks!
If you experienced this issue in v1.6.3, please try and report back on this bug. Thanks.
We've got a test all queued up. Just waiting for the rpms/debs to hit.
So I can confirm that the mentioned bug is fixed with 1.6.4. great, thanks.
Closing. Issue is fixed in release
Hi I'm facing this error, I'm using ansible to deploy kubernetes, so it brings the version: 1.5.2 , is their a way to resolve this issue without changing the release ? Thanks.
this specific bug was introduced with 1.6.3 and fixed with 1.6.4. Please open a separate issue with more details about your issue
ok thank u

BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): root kubectl version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-10T15:48:59Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.3+coreos.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-11T18:15:57Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: OS (e.g. from /etc/os-release): root cat /etc/os-release NAME="Ubuntu" VERSION="16.04.2 LTS (Xenial Xerus)" ID=ubuntu IDLIKE=debian PRETTYNAME="Ubuntu 16.04.2 LTS" VERSIONID="16.04" HOMEURL="" SUPPORTURL="" BUGREPORTURL="" VERSIONCODENAME=xenial UBUNTUCODENAME=xenial Kernel (e.g. ): root uname -a Linux ginger 4.4.0-78-generic -Ubuntu SMP Thu Apr 27 15:29:09 UTC 2017 x8664 x8664 x8664 GNU/Linux What happened: Updated latest Ubuntu packages from to Kubelet v1.6.3. Updated to Hyperkube v1.6.3.

Applied deployment from that also creates the following ConfigMaps: apiVersion: v1 kind: ConfigMap metadata: name: salt-syndic-master-config data: master: | autoaccept: True fileserverbackend: git gitfsremotes: extpillar: git: master syndicmaster: gitfssaltenv: kube: ref: master minionid: | playground.cluster.awesome- The pods of deployment are not able to mount the data from the configmaps as file using the following volumeMounts: volumeMounts: mountPath: /etc/salt/master name: salt-syndic-master-config subPath: master Instead I get the following errors from : May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] failed to mkdir:/var/lib/kubelet/pods/-3ae8-11e7-ade6- May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] container start failed: mkdir /var/lib/kubelet/pods/-3ae8-11e7-ade6- not a directory: Generate Container Config Failed May 17 13:56:44 ginger kubelet[]: E0517 13:56:44. ] Error syncing pod -3ae8-11e7-ade6- ("salt-syndic--kfj8w_awesome(-3ae8-11e7-ade6-)"), skipping: failed to "StartContainer" for "salt-syndic" with mkdir /var/lib/kubelet/pods/-3ae8-11e7-ade6- not a directory: "Generate Container Config Failed" Looking into shows that is a symlink to the which contains the correct data. Any help or tipps for further investigations are appreciated.
Duplicate of ...