Change:
pointed out that for scheduling and kubelet admission, we should consider resources of terminated-but-not-yet-deleted pods to still be in use, because it may take an arbitrary amount of time for the system to deallocate resources. We didn't check to see if this is currently how it is implemented or not. cc/ /sig scheduling
cc/
For terminated-but-not-yet-deleted & phase is case, the Pod is still in scheduler's cache, so it's handled well in scheduler; the Pod will be removed from cache when it's deleted. But in kubelet, it filtered terminated-but-not-yet-deleted out () when using scheduler.predicates. The option in my mind is to ~pass all Pods (including terminated-but-not-yet-deleted) to each admin, and let each admin decide whether filter terminated-but-not-yet-deleted out~ only filter out finished (Failed or Succeed) pods for Admits.
In -, it already consider by . So both scheduler & kubelet handles terminated-but-not-yet-deleted pods well.
Great, thanks for looking into it!
should this issue be closed then?
I think so.
Just to clarify: Kubelet correctly accounts the resources used by terminated-but-not-yet-deleted pods, i.e, kubelet does not count the resource usage of those pods towards total usage on the node. Scheduler, however, does not do the same. Scheduler waits for a pod to be deleted, and then it deducts its resource request from the associated node total usage. Upon deletion of a pod, scheduler is notified and acts: So, scheduler and kubelet behaviors are not exactly the same. I am not sure if scheduler can receive an event when a pod's container status changes. If it could, we could make the logic the same as that of Kubelet.
I guess the current behavior is OK because it is conservative (scheduler considers the resources to be in use even though kubelet considers them free). And actually it seems that scheduler is more "correct" than kubelet, since the resources really aren't usable until the pod is deleted, right? (IIUC that was Vish's original point)
I agree. The current behavior is correct, but a bit conservative. Basically, the scheduler side is more conservative and waits until a pod is deleted from the cache before considering its resources released. That said, I believe resources of terminated-but-not-deleted pods are usable on nodes (should be verified). So, the current behavior is a bit less than ideal in terms of efficiency, but AFAIK the duration between termination and deletion is only a few seconds. So, it may not be worth optimizing.
I thought was saying the opposite (see my first entry in this issue). Maybe he can clarify.

Discussion:
return pullSecrets } <del> // Returns true if pod is in the terminated state ("Failed" or "Succeeded"). </del> <ins> // podIsTerminated returns true if pod is in the terminated state ("Failed" or "Succeeded"). </ins> func (kl *Kubelet) podIsTerminated(pod *v1.Pod) bool { <del> var status v1.PodStatus </del> // Check the cached pod status which was set after the last sync. status, ok := kl.statusManager.GetPodStatus(pod.UID) if !ok {

defer testKubelet.Cleanup() kubelet := testKubelet.kubelet pods := newTestPods(5) <ins> now := metav1.NewTime(time.Now()) </ins> pods[0].Status.Phase = v1.PodFailed pods[1].Status.Phase = v1.PodSucceeded <ins> // The pod is terminating, should not filter out. </ins> pods[2].Status.Phase = v1.PodRunning <ins> pods[2].DeletionTimestamp = &now pods[2].Status.ContainerStatuses = []v1.ContainerStatus{ {State: v1.ContainerState{ Running: &v1.ContainerStateRunning{ StartedAt: now, }, }}, } </ins> pods[3].Status.Phase = v1.PodPending <ins> pods[4].Status.Phase = v1.PodRunning </ins> expected := []*v1.Pod{pods[2], pods[3], pods[4]} kubelet.podManager.SetPods(pods)