Change:
/kind bug What happened: I ran kubelet from HEAD and tried to make a normal cluster using kubeadm. I saw errors like these in the kubelet's log: (I think is the cause) What you expected to happen: No panic How to reproduce it (as minimally and precisely as possible): kubeadm the normal way kubelet, kubeadm from CI HEAD and a cluster like normal crashes Seems related to
cc I think the only thing we have to do here is to check for nil before trying to dereference the pointer Adding the v1.8 milestone to this bug to make we track it in a release-blocking sense, this is probably a release blocker. cc
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/node sig/storage: Issue will be escalated to these SIGs if needed. priority/critical-urgent: Never automatically move out of a release milestone; continually escalate to contributor and SIG through all available channels. kind/bug: Fixes a bug discovered during the current release. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
Since we've already set default value for , the value is a non-nil pointer. No obvious errors found in are related to .
Yeah, but somehow it seems to be nil here... otherwise it wouldn't have paniced
Before we merge, we need to understand how it got to be nil. It's supposed to be defaulted.
Absolutely. has a prototype of how it could be nil now in , let's verify if it fixes the issue...
I think volume manager reconstructs the specs from reading the directories on the disk and these specs do not have the defaults applied. would know better
Using api machinery validation without accompanying api machinery decoding/defaulting is dangerous... many paths make assumptions that the objects they are given have passed through standard decoding/defaulting
Agreed with both and above. can you assign this to the owner of the volume manager in sig-storage? cc/
The root cause is apiserver version < kubelet version, as I stated in .
Even with an old api server, it concerns me that there is apparently a code path that causes the kubelet to run validation on internal objects that did not go through a defaulting pass as part of being converted from v1 objects to internal objects.
This issue is being discussed on , see the latest on that PR. At the moment, it appears to be caused by kubeadm using a newer version of the k8s client then then the server. Awaiting word from and If that is the case, we should be able to close this bug. If not, we can merge as a temporary mitigation to unblock release.
Do you mean kubelet? We explicitly support clients being ahead of the server by a minor version.
Also, your comment doesn't address concern above.
Yes, referencing this discussion (): liggitt commented 17 hours ago Even with an old api server, it concerns me that there is apparently a code path that causes the kubelet to run validation on internal objects that did not go through a defaulting pass as part of being converted from v1 objects to internal objects. I don't think that is the case. See comment here: Based on that discussion, it looks like volume reconstruction was a red herring. In general volume reconstruction creates internal objects that are only used internally, they are not run through validation.
Still awaiting response on CC
Volume reconstruction is not entirely guilt free here: it constructs an internal object, does not run the object through validation or defaulting logic, so if a new field is introduced (like hostpath type in this case), the default value is not set resulting in issues like this one. To handle this case gracefully I'm approving PR For Q4 (1.9) we will follow up on this and see what we can do to handle this more gracefully. In general, volume reconstruction is a thorny bit of code and deserves a thorough redesign.
I have filed an issue related to internal volume reconstruction process during kublet restarts. The new proposed approach is very different from the current design. Will work on it in detail for 1.9
I got the same bug this is log Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: /usr/local/go/src/runtime/asmamd64.s:2361 Oct 24 14:45:27 node-23-40 kubelet[]: E1024 14:45:27. ] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: /usr/local/go/src/runtime/asmamd64.s:573 Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]: Oct 24 14:45:27 node-23-40 kubelet[]:
what version did you see the panic on?
kubectl version Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.2", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-09-19T07:01:43Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0-168+", GitCommit:"", GitTreeState:"clean", BuildDate:"2018-08-25T21:05:52Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
What flex driver are you using? It looks like it's not returning any capabilities. Here's the panic line: cc should we fallback to default capabilities if the driver doesn't return any?
This stacktrace is unrelated to the original failure. Can you open a new issue for this?
We already default it, unless the plugin overrides it with nil:

Discussion:
func validateHostPathType(hostPathType *api.HostPathType, fldPath *field.Path) field.ErrorList { allErrs := field.ErrorList{} <del> if !supportedHostPathTypes.Has(string(*hostPathType)) { </del> <ins> if hostPathType != nil && !supportedHostPathTypes.Has(string(*hostPathType)) { </ins> allErrs = append(allErrs, field.NotSupported(fldPath, hostPathType, supportedHostPathTypes.List())) }

} path := hostPathVolumeSource.Path <ins> pathType := new(v1.HostPathType) if hostPathVolumeSource.Type == nil { *pathType = v1.HostPathUnset } else { pathType = hostPathVolumeSource.Type } </ins> return &hostPathMounter{ <del> hostPath: &hostPath{path: path, pathType: hostPathVolumeSource.Type, containerized: opts.Containerized}, </del> <ins> hostPath: &hostPath{path: path, pathType: pathType, containerized: opts.Containerized}, </ins> readOnly: readOnly, }, nil }