Change:
What happened: A deployment is created that uses a after the cluster master is created but before the cluster worker nodes are fully initialized. During the initialization of the worker nodes, the nodes are temporarily available for scheduling without the necessary label to match the deployment's node selector. Depending on how long the nodes are available for scheduling without the necessary node labels for the deployment, the scheduler will start to spam with cluster with pods in status. This spamming stops once the worker nodes are fully initialized and the pods are scheduled successful. The exact timing of all of this is TBD. What you expected to happen: Pods in status are annoying and require manual cleanup. It would be preferred that the scheduler avoid this an simply issue warning events. How to reproduce it (as minimally and precisely as possible): See issue description. Anything else we need to know?: No. Environment: Kubernetes version (use ): 1.18.3 Cloud provider or hardware configuration: IBM Cloud Kubernetes Service OS (e.g: ): Ubuntu 18.04.4 LTS Kernel (e.g. ): 4.15.0-96-generic Install tools: Managed service install Network plugin and version (if this is a network-related bug): N/A Others: N/A
Related to and
/sig scheduling
by spamming are you referring to scheduler logs? also, why are the nodes not labelled from the get go?
Just to confirm, all the pods in question are still in Pending state, yes? If the pods are in pending state, all of them will get scheduled eventually once the nodes get their labels.
The pods are not in state. They are actually in state. The pods eventually get scheduled but we are left with anywhere from a few to dozens of these state pods. Also, we are looking at labeling the nodes via kubelet flags to see if that helps.
/remove-kind bug
After we add some retry logic and wait time in this problem can be solved: As you can see this function returns initial node as the first request failed.
I also get hit by this, when single controller node gets rebooted and scheduler starts as static container before kubelet becomes ready. Can be cleaned up with:
/sig node
the spam cycle is this loop: "pod create, pod schedule, worker reject, pod failed" -- we don't have a lot of protection in higher level workload controllers to handle pods that fail at kubelet admission time versus pods that just dont create due to quota/admission. in general, is the spam causing any other stability issues? is pod gc pruning the pods?
I wonder if we can do a trick in node lister to mitigate any race condition here...
we could maybe do something similar to to ensure node lister has synced...
hacking on option to ensure if kubelet has a node lister with a valid kube client, we wait for it to sync at least once here which should mitigate this issue. see:
We noticed the problem on the OLM pod which is managed by the OLM operator. Right or wrong, the operator does not like seeing the pod go to status and no longer reconciles the pod. The end result is that manual intervention (i.e. pod deletion) is required to fix the problem.
sure... i would have thought the pod is failed in that instance and reached a terminal state. either way, i am looking at what i can do to mitigate this.
The problem persists in our environment as well, and the workaround cited above will not do anymore. Is there something in the works for this one?
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale

Discussion:
// Max amount of time to wait for the container runtime to come up. maxWaitForContainerRuntime = 30 * time.Second <ins> // Max amount of time to wait for node list/watch to initially sync maxWaitForAPIServerSync = 10 * time.Second </ins> // nodeStatusUpdateRetry specifies how many times kubelet retries when posting node status failed. nodeStatusUpdateRetry = 5

serviceHasSynced = func() bool { return true } } <del> nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) </del> <ins> var nodeHasSynced cache.InformerSynced var nodeLister corelisters.NodeLister </ins> if kubeDeps.KubeClient != nil { <del> fieldSelector := fields.Set{api.ObjectNameField: string(nodeName)}.AsSelector() nodeLW := cache.NewListWatchFromClient(kubeDeps.KubeClient.CoreV1().RESTClient(), "nodes", metav1.NamespaceAll, fieldSelector) r := cache.NewReflector(nodeLW, &v1.Node{}, nodeIndexer, 0) go r.Run(wait.NeverStop) </del> <ins> kubeInformers := informers.NewSharedInformerFactoryWithOptions(kubeDeps.KubeClient, 0, informers.WithTweakListOptions(func(options *metav1.ListOptions) { options.FieldSelector = fields.Set{api.ObjectNameField: string(nodeName)}.String() })) nodeLister = kubeInformers.Core().V1().Nodes().Lister() nodeHasSynced = func() bool { if kubeInformers.Core().V1().Nodes().Informer().HasSynced() { klog.Infof("kubelet nodes sync") return true } klog.Infof("kubelet nodes not sync") return false } kubeInformers.Start(wait.NeverStop) klog.Info("Kubelet client is not nil") } else { // we dont have a client to sync! nodeIndexer := cache.NewIndexer(cache.MetaNamespaceKeyFunc, cache.Indexers{}) nodeLister = corelisters.NewNodeLister(nodeIndexer) nodeHasSynced = func() bool { return true } klog.Info("Kubelet client is nil") </ins> } <del> nodeLister := corelisters.NewNodeLister(nodeIndexer) </del> // TODO: get the real node object of ourself, // and use the real node name and UID.

serviceLister:                           serviceLister, serviceHasSynced:                        serviceHasSynced, nodeLister:                              nodeLister, <ins> nodeHasSynced:                           nodeHasSynced, </ins> masterServiceNamespace:                  masterServiceNamespace, streamingConnectionIdleTimeout:          kubeCfg.StreamingConnectionIdleTimeout.Duration, recorder:                                kubeDeps.Recorder,

serviceHasSynced cache.InformerSynced // nodeLister knows how to list nodes nodeLister corelisters.NodeLister <ins> // nodeHasSynced indicates whether nodes have been sync'd at least once. // Check this before trusting a response from the node lister. nodeHasSynced cache.InformerSynced </ins> // a list of node labels to register nodeLabels map[string]string

"io/ioutil" "net" "path/filepath" <ins> "time" </ins> cadvisorapiv1 "github.com/google/cadvisor/info/v1" cadvisorv2 "github.com/google/cadvisor/info/v2"

v1 "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/types" <ins> "k8s.io/apimachinery/pkg/util/wait" </ins> "k8s.io/kubernetes/pkg/kubelet/cm" "k8s.io/kubernetes/pkg/kubelet/config" kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"

if kl.kubeClient == nil { return kl.initialNode(context.TODO()) } <ins> // if we have a valid kube client, we wait for initial lister to sync if !kl.nodeHasSynced() { err := wait.PollImmediate(time.Second, maxWaitForAPIServerSync, func() (bool, error) { return kl.nodeHasSynced(), nil }) if err != nil { return nil, fmt.Errorf("nodes have not yet been read at least once, cannot construct node object") } } </ins> return kl.nodeLister.Get(string(kl.nodeName)) }

// zero capacity, and the default labels. func (kl *Kubelet) getNodeAnyWay() (*v1.Node, error) { if kl.kubeClient != nil { <del> if n, err := kl.nodeLister.Get(string(kl.nodeName)); err == nil { </del> <ins> if n, err := kl.GetNode(); err == nil { </ins> return n, nil } }

kubelet.masterServiceNamespace = metav1.NamespaceDefault kubelet.serviceLister = testServiceLister{} kubelet.serviceHasSynced = func() bool { return true } <ins> kubelet.nodeHasSynced = func() bool { return true } </ins> kubelet.nodeLister = testNodeLister{ nodes: []*v1.Node{ {