Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: If we delete node first, then remove pods on this node, the scheduler will crash in function What you expected to happen: Not crashed How to reproduce it (as minimally and precisely as possible): some pods on node A delete node A delete pods on node A Anything else we need to know?: Environment: Kubernetes version (use ): 1.16-1.18, even master Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig scheduling
can you show the core dump please of this crash?
/assign
I patched the unit test shown in into the , and branches. None crashed. I'll await for the crash dump.
Here is the fresh dump.
and which version was this?
This is master branch. And I've tested it from 1.16-1.18, it can be reproduced easily.
Thanks a lot. I was able to finally reproduce in unit tests. The scenario is: a node with at least 2 pods. node. pod1. In unit test, this is first an update to set the termination timestamp. No issues up to this point. pod2. Again, first an update to set the termination timestamp. Here it crashes. I'm working on a fix.
It looks like this is not an issue in 1.17 and below, because we preserved a placeholder node where the operations occur. do you have stacktraces for 1.17 and 1.16?
Yes, it seems only be found at 1.18. :smile:

Discussion:
// Assumes that lock is already acquired. func (cache *schedulerCache) updatePod(oldPod, newPod *v1.Pod) error { <ins> if _, ok := cache.nodes[newPod.Spec.NodeName]; !ok { // The node might have been deleted already. // This is not a problem in the case where a pod update arrives before the // node creation, because we will always have a create pod event before // that, which will create the placeholder node item. return nil } </ins> if err := cache.removePod(oldPod); err != nil { return err }

// Create a few pods for tests. pods := []*v1.Pod{} <del> for i := 0; i < 10; i++ { </del> <ins> for i := 0; i < 20; i++ { </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name:      fmt.Sprintf("test-pod%v", i),

UID:       types.UID(fmt.Sprintf("test-puid%v", i)), }, Spec: v1.PodSpec{ <del> NodeName: fmt.Sprintf("test-node%v", i), </del> <ins> NodeName: fmt.Sprintf("test-node%v", i%10), </ins> }, } pods = append(pods, pod)

} } } <ins> removePod := func(i int) operation { return func() { if err := cache.RemovePod(pods[i]); err != nil { t.Error(err) } } } </ins> removePodWithAffinity := func(i int) operation { return func() { if err := cache.RemovePod(podsWithAffinity[i]); err != nil {

expected: []*v1.Node{nodes[6], nodes[5], nodes[2], nodes[0]}, }, { <del> name: "Remove non-existing node", operations: []operation{ addNode(0), addNode(1), updateSnapshot(), }, expected: []*v1.Node{nodes[1], nodes[0]}, }, { </del> name: "Update some nodes", operations: []operation{ addNode(0), addNode(1), addNode(5), updateSnapshot(), updateNode(1),

expected: []*v1.Node{nodes[0], nodes[4], nodes[2]}, }, { <del> name: "Remove pod from non-existing node", </del> <ins> name: "Add pod before its node", operations: []operation{ addNode(0), addPod(1), updatePod(1), addNode(1), }, expected: []*v1.Node{nodes[1], nodes[0]}, }, { name: "Remove node before its pods", </ins> operations: []operation{ <del> addNode(0), addPod(0), addNode(2), updateSnapshot(), </del> <ins> addNode(0), addNode(1), addPod(1), addPod(11), removeNode(1), updatePod(1), updatePod(11), removePod(1), removePod(11), </ins> }, <del> expected: []*v1.Node{nodes[2], nodes[0]}, </del> <ins> expected: []*v1.Node{nodes[0]}, </ins> }, { name: "Add Pods with affinity",