Change:
How to reproduce it (as minimally and precisely as possible): pre-requisites: a multi-az cluster, a deployment with 1 pod in each az. a with and . the node(s) in 1 az. the deployment's pod(s) from the cordoned az. What happened: A new pod was created and scheduled in a form that What you expected to happen: I expected that a new pod will be created and be "stuck" in status as it shouldn't have nowhere be scheduled. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g: ):Kernel (e.g. ): 4.19.84-coreos Others: I created a with how I reproduce the bug
/sig scheduling
cc/
Thanks for raising this issue. I feel like it's a general timing issue - when you delete a Pod, that Pod may still exist for a very short time (in graceful deletion period), and at the same time, the controller backfills a Pod for it. So when scheduling tries to schedule the backfill Pod, it may read the cluster as: [1, 1, 1(deleting)]. To confirm this, can you try deleting the Pod using , and see if the problem goes away?
Yes sure I will try that. It sounds reasonable, thats why in the first demonstration I deleted all the pods and waited for all of them to completely disappear. Nevertheless, this is still something that I believe needs to be fixed. The pod's evaluation should take in consideration pods that are in status and don't count them as pods.
I do believe we exclude the Pods. (if not, it should be a serious regression). What concerns me is that at that moment, scheduler may treats that (deleting) Pod as running in its cache. I will try to debug into this.
Yep, this time the new pod was in status until I uncordoned the relevant nodes.

Discussion:
wErr:  fmt.Errorf("error while running score plugin for pod "2": %+v", errPrioritize), }, { <del> name: "test even pods spread predicate - 2 nodes with maxskew=1", </del> <ins> name: "test podtopologyspread plugin - 2 nodes with maxskew=1", </ins> registerPlugins: []st.RegisterPluginFunc{ st.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), st.RegisterPluginAsExtensions(

wErr:          nil, }, { <del> name: "test even pods spread predicate - 3 nodes with maxskew=2", </del> <ins> name: "test podtopologyspread plugin - 3 nodes with maxskew=2", </ins> registerPlugins: []st.RegisterPluginFunc{ st.RegisterQueueSortPlugin(queuesort.Name, queuesort.New), st.RegisterPluginAsExtensions(

schedulerapi.DefaultPercentageOfNodesToScore, false) cache.UpdateSnapshot(s.(*genericScheduler).nodeInfoSnapshot) return s.(*genericScheduler) } func TestFindFitAllError(t *testing.T) {

matchTotal := int32(0) // nodeInfo.Pods() can be empty; or all pods don't fit for _, existingPod := range nodeInfo.Pods() { <del> if existingPod.Namespace != pod.Namespace { </del> <ins> // Bypass terminating Pod (see #87621). if existingPod.DeletionTimestamp != nil || existingPod.Namespace != pod.Namespace { </ins> continue } if constraint.selector.Matches(labels.Set(existingPod.Labels)) {

}, }, { <del> name: "add a pod with mis-matched namespace doesn't change topologyKeyToMinPodsMap", </del> <ins> name: "add a pod in a different namespace doesn't change topologyKeyToMinPodsMap", </ins> preemptor: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(),

}, }, { <del> name: "existing pods with mis-matched namespace doesn't count", </del> <ins> name: "existing pods in a different namespace do not count", </ins> pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "zone", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(),

"node-y": false, }, }, <ins> { name: "terminating Pods should be excluded", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "node", hardSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("node", "node-b").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a").Node("node-a").Label("foo", "").Terminating().Obj(), st.MakePod().Name("p-b").Node("node-b").Label("foo", "").Obj(), }, fits: map[string]bool{ "node-a": true, "node-b": false, }, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

// <matchSum> indicates how many pods (on current node) match the <constraint>. matchSum := int64(0) for _, existingPod := range nodeInfo.Pods() { <ins> // Bypass terminating Pod (see #87621). if existingPod.DeletionTimestamp != nil || existingPod.Namespace != pod.Namespace { continue } </ins> if c.selector.Matches(labels.Set(existingPod.Labels)) { matchSum++ }

{Name: "node-x", Score: 100}, }, }, <ins> { name: "existing pods in a different namespace do not count", pod: st.MakePod().Name("p").Label("foo", ""). SpreadConstraint(1, "node", softSpread, st.MakeLabelSelector().Exists("foo").Obj()). Obj(), existingPods: []*v1.Pod{ st.MakePod().Name("p-a1").Namespace("ns1").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-a2").Node("node-a").Label("foo", "").Obj(), st.MakePod().Name("p-b1").Node("node-b").Label("foo", "").Obj(), st.MakePod().Name("p-b2").Node("node-b").Label("foo", "").Obj(), }, nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("node", "node-b").Obj(), }, want: []framework.NodeScore{ {Name: "node-a", Score: 100}, {Name: "node-b", Score: 50}, }, }, { name: "terminating Pods should be excluded", pod: st.MakePod().Name("p").Label("foo", "").SpreadConstraint( 1, "node", softSpread, st.MakeLabelSelector().Exists("foo").Obj(), ).Obj(), nodes: []*v1.Node{ st.MakeNode().Name("node-a").Label("node", "node-a").Obj(), st.MakeNode().Name("node-b").Label("node", "node-b").Obj(), }, existingPods: []*v1.Pod{ st.MakePod().Name("p-a").Node("node-a").Label("foo", "").Terminating().Obj(), st.MakePod().Name("p-b").Node("node-b").Label("foo", "").Obj(), }, want: []framework.NodeScore{ {Name: "node-a", Score: 100}, {Name: "node-b", Score: 0}, }, }, </ins> } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) {

return p } <ins> // Terminating sets the inner pod's deletionTimestamp to current timestamp. func (p *PodWrapper) Terminating() *PodWrapper { now := metav1.Now() p.DeletionTimestamp = &now return p } </ins> // ZeroTerminationGracePeriod sets the TerminationGracePeriodSeconds of the inner pod to zero. func (p *PodWrapper) ZeroTerminationGracePeriod() *PodWrapper { p.Spec.TerminationGracePeriodSeconds = &zero