Change:
The scheduler has a for the bind operation to succeed; if we don't get a response within 30s, the in-memory assignment of pod to node in the . A race condition will happen in the follow case: 1 is assigned to a node, scheduler cache is updated with the assignment, bind operation issued to apiserver. the apiserver is under huge pressure, bind takes more than 30s, scheduler expires the cached pod-to-node assignment. eventually succeeds, but because the apiserver is under huge pressure, the pod update with the node name takes a long time to propagate to the scheduler. the pod update took a long time to propagate and the cache entry expired, the scheduler is not aware that the assignment actually happened, and so it had no problem assigning a second pod to the same node that would otherwise not fit if the scheduler was aware that the first pod was eventually assigned to the node. On the scheduler side, what we need to do is make the 30s longer for large clusters, and ideally adaptable to cluster state. /sig scheduling No race conditions. Create a large scale cluster. No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
IIRC, the 30s timeout only starts running after we receive a 200 response for the binding. But yes, if it takes longer than 30s to receive the update, we would invalidate the cache. I don't know if there is particular reason to invalidate the cache at all. I suppose it was meant to guard against missing Pod deletion events. We need to improve that regardless. I would be in favor of dropping the timeout altogether.
ok, so the deadline is effectively for receiving the update. Perhaps making it longer, like 15min, is good first step before completely removing it just so we can easily revert in case something else goes wrong.
I sent to increase the timeout to 15min. We should backport this.
We should double check the history, but I think that in the past we were: asynchronously sending bind request updating the cache before sending the request and then the timeout was actually protecting us from the binding request actually not succeeding. So I agree with Aldo - that we should double check if this timeout is at all needed currently.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
We are probably good to remove this timeout altogether at this point. /remove-lifecycle stale
/good-first-issue
I see this issue was labeled as "good first issue" and I would like to start working on it!
I would like to take this task
Plz assign to yourself to notify that you"re working on this.
/assign
/assign
done. thanks!
/assign
Could you please guide for this Issue are you suggesting we should not check cache before assigning the pod and need other way around?
FTR, I'm considering assigned to this issue, as they were the first to get assigned. You can see that in we increased the timeout. We want to get rid of the timeout. Then, we can just get rid of the logic that uses the timeout to remove some pods from the cache.
/reopen We still need to remove the code in 1.27
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/unassign
one more release before we can remove this code.
do you have plan to cherry pick for previous versions, e.g 1.23, 1.24?
did you mean 1.13 and 1.14?
oh, sorry my bad, I meant for 1.23 and 1.24
It is fixed in 1.23 , so it would be fixed in 1.24 too.
my understanding is the previous fix to increase the duration only reduce the chance of issue, but not completely avoid it, that's why the other 2 fixes and are needed, right?
Hi, is this issue still available?
yes, but since the likelihood that an issue happens is very low, I don't think it's worth the risk of cherry-picking at this point.
According to , we can remove the TTL code entirely in 1.27 I'll give the chance to to confirm if they can still do this. In the meantime: /unassign /unassign /remove-help
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale /unassign as we haven't heard from them.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
I want to help fix this /assign
Thanks. What is left is to remove the timeout logic altogether.
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen remove the expire logic
You can't reopen an issue/PR unless you authored it or you are a collaborator. <detailsIn response to : /reopen remove the expire logic Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I think we can leave this closed. Entirely removing the timeouts has proven to be close to impossible without introducing regressions. We might need a few more iterations. Maybe we can reconsider when scheduling hints graduates to Stable.

Discussion:
"pod", klog.KObj(ps.pod)) continue } <del> if now.After(*ps.deadline) { </del> <ins> if cache.ttl != 0 && now.After(*ps.deadline) { </ins> klog.InfoS("Pod expired", "pod", klog.KObj(ps.pod)) if err := cache.removePod(ps.pod); err != nil { klog.ErrorS(err, "ExpirePod failed", "pod", klog.KObj(ps.pod))

const ( // Duration the scheduler will wait before expiring an assumed pod. // See issue #106361 for more details about this parameter and its value. <del> durationToExpireAssumedPod = 15 * time.Minute </del> <ins> durationToExpireAssumedPod = 0 * time.Minute </ins> ) // ErrNoNodesAvailable is used to describe the error that no nodes available to schedule pods.