Code change:
serviceHasSynced = func() bool { return true } } <del> nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) </del> <ins> var nodeHasSynced cache.InformerSynced var nodeLister corelisters.NodeLister </ins> if kubeDeps.KubeClient != nil { <del> fieldSelector := fields.Set{api.ObjectNameField: string(nodeName)}.AsSelector() nodeLW := cache.NewListWatchFromClient(kubeDeps.KubeClient.CoreV1().RESTClient(), "nodes", metav1.NamespaceAll, fieldSelector) r := cache.NewReflector(nodeLW, &v1.Node{}, nodeIndexer, 0) go r.Run(wait.NeverStop) </del> <ins> kubeInformers := informers.NewSharedInformerFactoryWithOptions(kubeDeps.KubeClient, 0, informers.WithTweakListOptions(func(options *metav1.ListOptions) { options.FieldSelector = fields.Set{api.ObjectNameField: string(nodeName)}.String() })) nodeLister = kubeInformers.Core().V1().Nodes().Lister() nodeHasSynced = func() bool { if kubeInformers.Core().V1().Nodes().Informer().HasSynced() { klog.Infof("kubelet nodes sync") return true } klog.Infof("kubelet nodes not sync") return false } kubeInformers.Start(wait.NeverStop) klog.Info("Kubelet client is not nil") } else { // we dont have a client to sync! nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) nodeLister = corelisters.NewNodeLister(nodeIndexer) nodeHasSynced = func() bool { return true } klog.Info("Kubelet client is nil") </ins> } <del> nodeLister := corelisters.NewNodeLister(nodeIndexer) </del> // TODO: get the real node object of ourself, // and use the real node name and UID.

Related discussion:
text: What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready.