Change:
What happened: pod can't be removed from scheduler cache when be deleted, logs in kube-scheduler is below: I1128 16:41:50. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:50. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:53. ] lock is held by shaphicpra1490764ccb0c5-aaa1-435d-a4b4- and has not yet expired I1128 16:41:53. ] failed to acquire lease kube-system/kube-scheduler I1128 16:41:57. ] successfully acquired lease kube-system/kube-scheduler I1128 16:41:57. ] caches populated E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:12. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it What you expected to happen: pod in scheduler cache can be deleted successfully when pod is deleted How to reproduce it (as minimally and precisely as possible): main steps to reproduce this bug is these: scheduler run in leader-elect=true modeschedulerA is the leader scheduler, and schedulerB is an assistant. pod1 with "namespace1/name1" and it runs on Node1 down the network between schedulerB and kube-apiserver pod1 and create pod2 with same name, ie "namespace1/name1" and pod2 been assigned to Node1 by schedulerA again. off schedulerA and recover the network between schedulerB and kube-apiserver, and then schedulerB become the leader scheduler. pod2. this time log in schedulerB will print this: E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated E1128 16:42:20. ] scheduler cache RemovePod failed: pod -decb-4abd-a1d3- is not found in scheduler cache, so cannot be removed from it this time, if I create a new pod3 with podAntiAffinity to pod1. the result is pod3 can't been assigned to Node1 because pod1 is still in schedulerB's scheduler cache. Anything else we need to know?: the reason of this bug is below: since PR(), podKey in scheduler cache use pod's UID instead of namespace/name however, the key in SharedIndexInformer still use namespace/name in this upper case, delete the pod and create a same name pod may be delivered as one update notification.( see disscusion ) scheduler received pod update notification and then print below log. E1128 16:42:11. ] scheduler cache UpdatePod failed: pod -decb-4abd-a1d3- is not to scheduler cache, so cannot be updated Environment: Kubernetes version (use ): Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"", GitTreeState:"clean", BuildDate:"2019-06-19T16:40:16Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"} Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig release
/sig scheduling
Since has been marked resolved (fix available in 1.17), does that mean this issue is expected to go away in 1.17? Could you please give a try?
, fix for just resolved the case add event may be lost after deletion event. It doesn't solve the problem that a delete followed by an add may be delivered as one update notification and it's thought normal from MikeSpreitzer's comment. The main cause of this problem is scheduler cache use Pod UID as key while sharedInformer use namespace/name. Still, I also test with the latest kubernetes code. The bug of this issue reappeared.
Thanks I a comment in the PR, and a test is necessary to reproduce the issue and verify the fix.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale ping there is an unresolved comment in the PR, do you have a chance to take a look?

Discussion:
return } <ins> // A Pod delete event followed by an immediate Pod add event may be merged // into a Pod update event. In this case, we should invalidate the old Pod, and // then add the new Pod. if oldPod.UID != newPod.UID { sched.deletePodFromCache(oldObj) sched.addPodToCache(newObj) return } </ins> // NOTE: Updates must be written to scheduler cache before invalidating // equivalence cache, because we could snapshot equivalence cache after the // invalidation and then snapshot the cache itself. If the cache is

import ( "reflect" "testing" <ins> "time" </ins> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/resource" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" <ins> "k8s.io/kubernetes/pkg/scheduler/internal/cache" </ins> fakecache "k8s.io/kubernetes/pkg/scheduler/internal/cache/fake" <ins> "k8s.io/kubernetes/pkg/scheduler/internal/queue" </ins> ) func TestSkipPodUpdate(t *testing.T) {

}) } } <ins> func TestUpdatePodInCache(t *testing.T) { ttl := 10 * time.Second nodeName := "node" tests := []struct { name   string oldObj interface{} newObj interface{} }{ { name:   "pod updated with the same UID", oldObj: withPodName(podWithPort("oldUID", nodeName, 80), "pod"), newObj: withPodName(podWithPort("oldUID", nodeName, 8080), "pod"), }, { name:   "pod updated with different UIDs", oldObj: withPodName(podWithPort("oldUID", nodeName, 80), "pod"), newObj: withPodName(podWithPort("newUID", nodeName, 8080), "pod"), }, } for _, tt := range tests { t.Run(tt.name, func(t *testing.T) { stopCh := make(chan struct{}) defer close(stopCh) schedulerCache := cache.New(ttl, stopCh) schedulerQueue := queue.NewPriorityQueue(nil) sched := &Scheduler{ SchedulerCache:  schedulerCache, SchedulingQueue: schedulerQueue, } sched.addPodToCache(tt.oldObj) sched.updatePodInCache(tt.oldObj, tt.newObj) pod, err := sched.SchedulerCache.GetPod(tt.newObj.(*v1.Pod)) if err != nil { t.Errorf("Failed to get pod from scheduler: %v", err) } if pod.UID != tt.newObj.(*v1.Pod).UID { t.Errorf("Want pod UID %v, got %v", tt.newObj.(*v1.Pod).UID, pod.UID) } }) } } func withPodName(pod *v1.Pod, name string) *v1.Pod { pod.Name = name return pod } </ins>

func podWithID(id, desiredHost string) *v1.Pod { return &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ <del> Name:     id, UID:      types.UID(id), SelfLink: fmt.Sprintf("/api/v1/%s/%s", string(v1.ResourcePods), id), </del> <ins> Name: id, UID:  types.UID(id), </ins> }, Spec: v1.PodSpec{ NodeName:      desiredHost,

Name:              id, UID:               types.UID(id), DeletionTimestamp: &deletionTimestamp, <del> SelfLink:          fmt.Sprintf("/api/v1/%s/%s", string(v1.ResourcePods), id), </del> }, Spec: v1.PodSpec{ NodeName:      "",