Change:
k8s 1.6.3 just landed and broke a lot of our systems. The issue shows as: 2017-05-10 20:07:19. | NAME READY STATUS RESTARTS AGE 2017-05-10 20:07:19. | test-ceph-init-mon-hx3h1 0/1 mkdir /var/lib/kubelet/pods/-35bb-11e7-8ad3- not a directory 0 3m The issue is, we have subPath set to a file in a configmap to map it into the right place in our container. But the new code assumes subPath is a directory and always does a mkdir over it. This causes things to fail. The PR in question is:
cc /assign wongma7
Reiterating the mentions to trigger a notification: <detailsIn response to : cc /assign wongma7 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. I understand the commands that are listed . </details
Its also related to:
can you ensure your fix is picked to 1.6 branch, too?
yes, I will.
any chance it gets finally committed? our testing gates are all red because of this issue..
I brought it up in the sig-storage meeting today so priority was raised to critical-urgent and sig node was pinged. I guess we could make some more noise, I'll try finding people on slack
Would be great to get it out one way or another. I have heard other people also see this issue.
Status: it's merging to master right now & I just opened a cherry-pick for it into 1.6 I'm not sure what the procedure is for expediting a point release, but I suppose it would be enough if those other people +1 this issue, it may help the folks in charge judge the severity of the bug. and what's the procedure here? The context is that a bug was introduced with 1.6.3 that makes it so that certain pods (that mount a volume's file using the subPath field) fail to start. Thanks!
FYI, closed by mistake, sorry. We still need to track it 'til it's fixed in 1.6.x
I think an expedited release would be warranted if there are a large number of users who both (1) are affected by this issue, and (2) need something in 1.6.3 so rolling back to 1.6.2 is not good enough. I don't know how common it is for users to rely on this feature, so any signal we can get on that would be helpful.
this feature is used a lot by kolla-kubernetes project. Rolling back is problematic, our gating system works with centos and Ubuntu images, with centos we can go back to 1.6.2 but with Ubuntu, it looks like 1.6.2 binaries have been already removed. We use binaries which are provided by repo and not static files.
Perhaps we should add an e2e test case for subPath.
yeah, i have to agree with and on this one. our projects are very similar, and we're experiencing the same thing on our end w/v1.6.3.
After discussing with I am planning to expedite v1.6.4 to fix this. However, I will wait for the e2e test, which I believe is working on.
Keeping open until v1.6.4 ships with fix.
the e2e test PR is here: . They cover both this issue and the original one I was trying to fix, .
should kubernetes yank the binary packages for kubelet/kubeadm/etc that were released for 1.6.3 (debian/rpm/etc) - until 1.6.4 gets released? Because many of us who are just installing kubeadm/kubelet/etc with 1.6.* have found that things suddenly start break. Right now, its in the release note with a comment like . is not an alpha/beta feature, it's a core feature, and we just can't assume someone "may or may not" use it. Should this be rather more critical than that? 3 is a broken release and nobody should ever install it right?
as a workaround we compiled 1.6.3 kubelet with the one fix. If you are interested, please let me know I will share the link, I have it posted on GitHub.
thank you, we have pinned kubernetes versions for now... I can see that k8s 1.6.4-beta.1 is already being tagged. Great work from the team! Looks like k8s 1.6.4 will supersede 1.6.3 shortly. Thanks!
If you experienced this issue in v1.6.3, please try and report back on this bug. Thanks.
We've got a test all queued up. Just waiting for the rpms/debs to hit.
So I can confirm that the mentioned bug is fixed with 1.6.4. great, thanks.
Closing. Issue is fixed in release
Hi I'm facing this error, I'm using ansible to deploy kubernetes, so it brings the version: 1.5.2 , is their a way to resolve this issue without changing the release ? Thanks.
this specific bug was introduced with 1.6.3 and fixed with 1.6.4. Please open a separate issue with more details about your issue
ok thank u

Discussion:
"k8s.io/kubernetes/pkg/kubelet/status" kubetypes "k8s.io/kubernetes/pkg/kubelet/types" "k8s.io/kubernetes/pkg/kubelet/util/format" <ins> "k8s.io/kubernetes/pkg/util" </ins> "k8s.io/kubernetes/pkg/volume" "k8s.io/kubernetes/pkg/volume/util/volumehelper" "k8s.io/kubernetes/third_party/forked/golang/expansion"

hostPath = filepath.Join(hostPath, mount.SubPath) <del> // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } </del> <ins> if subPathExists, err := util.FileExists(hostPath); err != nil { glog.Errorf("Could not determine if subPath %s exists; will not attempt to change its permissions", hostPath) } else if !subPathExists { // Create the sub path now because if it's auto-created later when referenced, it may have an // incorrect ownership and mode. For example, the sub path directory must have at least g+rwx // when the pod specifies an fsGroup, and if the directory is not created here, Docker will // later auto-create it with the incorrect mode 0750 if err := os.MkdirAll(hostPath, perm); err != nil { glog.Errorf("failed to mkdir:%s", hostPath) return nil, err } </ins> <del> // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err </del> <ins> // chmod the sub path because umask may have prevented us from making the sub path with the same // permissions as the mounter path if err := os.Chmod(hostPath, perm); err != nil { return nil, err } </ins> } }