Code change:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Scheduler's pod informer has a selector that . This is incorrect. Scheduler must watch all the pods of the cluster, including those scheduled by other schedulers in order to update its cache of the state of the cluster. Otherwise, it will miss important events and the amount of available resources, pods on nodes, ... which are needed for scheduler to work correctly, will not be updated properly. /sig scheduling /assign
cc
For updating it cache, shouldn't the default scheduler rely on node info/status? Also its not very clear, what information the default scheduler needs from other pods (not to be scheduled by it) that it makes decision on? Just trying to understand the issue more clearly.
Scheduler watches informer events to update its cache. Node status is not enough to update all the information that scheduler caches. Scheduler must know what pods are placed on what nodes. For example, in order to check inter-pod affinity/anti-affinity it needs to know the placement of all the pods, including those scheduled by other schedulers.
But this information is present here: So it seems that your PR will now make sure to get the scheduled pods' information directly from the pod informer.
So when I check the history, I see it was here:
have we seen any specific issues due to this change (like between default scheduler and DS controller)?
Yes, but scheduler relies on its pod informers to keep this location up to date. That's exactly why it is important for the scheduler to receive updates for all the pods in the cluster. Yes. We missed the problem at the time. This is problematic only in clusters that use multiple schedulers. I initially thought that DS controller could cause issues as well, but DS controller does not set , so it must not cause an issue. We have seen instances of scheduler's missing certain events in cluster and as a result scheduler's cache getting stale, but I doubt if it is related to this particular issue. This only affects clusters with more than one scheduler. If you run the test that I have written in with the head, it will fail. That tests simulate a multi-scheduler scenario.

Related discussion:
text: t.Errorf("No PDB was deleted from the cache: %v", err) } } <ins> // TestSchedulerInformers tests that scheduler receives informer events and updates its cache when // pods are scheduled by other schedulers. func TestSchedulerInformers(t *testing.T) { // Initialize scheduler. context := initTest(t, "scheduler-informer") defer cleanupTest(t, context) cs := context.clientSet defaultPodRes := &v1.ResourceRequirements{Requests: v1.ResourceList{ v1.ResourceCPU:    *resource.NewMilliQuantity(200, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(200, resource.BinarySI)}, } defaultNodeRes := &v1.ResourceList{ v1.ResourcePods:   *resource.NewQuantity(32, resource.DecimalSI), v1.ResourceCPU:    *resource.NewMilliQuantity(500, resource.DecimalSI), v1.ResourceMemory: *resource.NewQuantity(500, resource.BinarySI), } type nodeConfig struct { name string res  *v1.ResourceList } tests := []struct { description         string nodes               []*nodeConfig existingPods        []*v1.Pod pod                 *v1.Pod preemptedPodIndexes map[int]struct{} }{ { description: "Pod cannot be scheduled when node is occupied by pods scheduled by other schedulers", nodes:       []*nodeConfig{{name: "node-1", res: defaultNodeRes}}, existingPods: []*v1.Pod{ initPausePod(context.clientSet, &pausePodConfig{ Name:          "pod1", Namespace:     context.ns.Name, Resources:     defaultPodRes, Labels:        map[string]string{"foo": "bar"}, NodeName:      "node-1", SchedulerName: "foo-scheduler", }), initPausePod(context.clientSet, &pausePodConfig{ Name:          "pod2", Namespace:     context.ns.Name, Resources:     defaultPodRes, Labels:        map[string]string{"foo": "bar"}, NodeName:      "node-1", SchedulerName: "bar-scheduler", }), }, pod: initPausePod(cs, &pausePodConfig{ Name:      "unschedulable-pod", Namespace: context.ns.Name, Resources: defaultPodRes, }), preemptedPodIndexes: map[int]struct{}{2: {}}, }, } for _, test := range tests { for _, nodeConf := range test.nodes { _, err := createNode(cs, nodeConf.name, nodeConf.res) if err != nil { t.Fatalf("Error creating node %v: %v", nodeConf.name, err) } } pods := make([]*v1.Pod, len(test.existingPods)) var err error // Create and run existingPods. for i, p := range test.existingPods { if pods[i], err = runPausePod(cs, p); err != nil { t.Fatalf("Test [%v]: Error running pause pod: %v", test.description, err) } } // Create the new "pod". unschedulable, err := createPausePod(cs, test.pod) if err != nil { t.Errorf("Error while creating new pod: %v", err) } if err := waitForPodUnschedulable(cs, unschedulable); err != nil { t.Errorf("Pod %v got scheduled: %v", unschedulable.Name, err) } // Cleanup pods = append(pods, unschedulable) cleanupPods(cs, t, pods) cs.PolicyV1beta1().PodDisruptionBudgets(context.ns.Name).DeleteCollection(nil, metav1.ListOptions{}) cs.CoreV1().Nodes().DeleteCollection(nil, metav1.ListOptions{}) } } </ins>