Change:
func initializeCSINode(host volume.VolumeHost) error { kvh, ok := host.(volume.KubeletVolumeHost) if !ok { <del> klog.V(4).Info("Cast from VolumeHost to KubeletVolumeHost failed. Skipping CSINodeInfo initialization, not running on kubelet") </del> <ins> klog.V(4).Info("Cast from VolumeHost to KubeletVolumeHost failed. Skipping CSINode initialization, not running on kubelet") </ins> return nil } kubeClient := host.GetKubeClient() if kubeClient == nil { <del> // Kubelet running in standalone mode. Skip CSINodeInfo initialization klog.Warning("Skipping CSINodeInfo initialization, kubelet running in standalone mode") </del> <ins> // Kubelet running in standalone mode. Skip CSINode initialization klog.Warning("Skipping CSINode initialization, kubelet running in standalone mode") </ins> return nil } <del> kvh.SetKubeletError(errors.New("CSINodeInfo is not yet initialized")) </del> <ins> kvh.SetKubeletError(errors.New("CSINode is not yet initialized")) </ins> go func() { defer utilruntime.HandleCrash() <ins> // First wait indefinitely to talk to Kube APIServer nodeName := host.GetNodeName() err := waitForAPIServerForever(kubeClient, nodeName) if err != nil { klog.Fatalf("Failed to initialize CSINode while waiting for API server to report ok: %v", err) } </ins> // Backoff parameters tuned to retry over 140 seconds. Will fail and restart the Kubelet // after max retry steps. initBackoff := wait.Backoff{

Discussion:
What happened: Upgraded from 1.16.6 to 1.17.2 and will no longer start on control-plane hosts. It fails with: What you expected to happen: to start, static pods to launch, to connect to in the static pod How to reproduce it (as minimally and precisely as possible): Have the , , , etc in the static pod manifest folder. Try to start the with the container images not preloaded on the host. Just in case it is needed: is used as the backend is run using Anything else we need to know?: This looks to be two things interacting poorly: 1- It takes a long time for static pods to be created for the first time when the can't talk to yet. 2- The new CSI migration code force terminates causing these log lines to appear in : It looks like introduced this new fatal error. Environment: Kubernetes version (use ): 1.17.2 Cloud provider or hardware configuration: Bare-metal OS (e.g: ): Fedora 31..1 (CoreOS preview)" Kernel (e.g. ): 5.3.12-300.fc31.x86_64 Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cluster-lifecycle
Actually, probably: /sig node
The referenced PR was in 1.14, so that interaction is not new. What is likely new is the size increase of hyperkube in 1.17 (xref ). When not pre-pulled, this can increase the time it takes to start hyperkube for the first time. Does the image pull make progress as the kubelet restarts/retries?
It might make sense for the initializeCSINode() method to: first wait to successfully talk to the kube apiserver at all, perhaps indefinitely after successfully talking to the kube apiserver, then try to register the CSINode object with backoff, and fail if it cannot do that successfully
/sig storage /remove-sig cluster-lifecycle
It doesn't seem to. Once the connection to containerd is closed, it cancels the context. Otherwise I would expect this to succeed eventually if the pull happened in the background. That sounds reasonable to me assuming it isn't blocking the static pod creation code. Do you happen to know why it takes a very long time for the static pod creation code to be called?

I would expect the pods to be created very soon after the kubelet is started.
Pulling the image and starting the processes for those pods happens immediately (from the containerd logs, you can see the image pull happening... it just doesn't complete in time before the independent kubelet logic to register the CSI node info times out).
Ah, yes. I just upgraded a couple of days ago so I didn't check whether I still saw the 2 minute delay in creating the "pod sandboxes" (this is before the pull even happens). It now is only a 23 second delay: containerd: kubelet:
ack, I like the idea Will work on a fix today and cherry-pick it as necessary