jiebi/CodeConvo huggingface dataset:
For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.
So instead of "pass all unscheduled Pods to movePodsToActiveOrBackoffQueue", I'd like to come up with a solution like: It's pretty rare to move pods depending on Unschedulable Pods' events. In your particular case, maybe a proactive call would behave more efficiently. It doesn't sound correlated, is it? As the check will just pre-verify the basic checks which are orthogonal with PodTopologySpread. In other words, it's impossible that a Pod failed can become schedulable eventually. Updated: The example is a valid point... Can you share some use-cases to move Pods upon node's deletion? In the before, I don't think we have any, until the last release the introduction of feature (as in that case deleting a node would impact the number of domains).
to come up with a solution like: If we do so, also, we need to provide the way to pass some functions to decide which Pods should pass to like for out-of-tree plugins? call would behave more efficiently. Ah~, that's a good idea :) Maybe like this (similar to ) user uses WhenUnsatisfiable: DoNotSchedule. Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses maxSkew: 1 and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. deletes NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue. So,,, ignoring some events for performance in scheduler side (without providing a way to change this) could be inconvenient for some plugins. What I thought in the above previous comment () is to define "which kind of resource's events should be ignored" in plugin side. (similar to 's idea? ) We can add pre filter functions in like: Then we can set any pre-filter functions for each plugin. And, we can define some default behavior for PreFilterFn and use them when is empty. So, for example, if most of the plugins are considered to prefer to ignore the event for Node that doesn't match nodeAffinity/nodeSelector, we can define this behavior as the default behavior.
What do you think about this idea?
Thanks. It makes sense to me. To summarize, the following cases doesn't properly re-queue relevant pods: Pod's add/update events -scheduled Pod's add/update/delete events 's add/update/delete events For 1 & 2, I've commented above with solutions. (1 needs core changes similar like podAffinity, 2 may leverage the Activate hook) For 3, with 's examples, I'm convinced this is some area we need to improve. One simple and stratightforward solution is to plumb a hook to let the plugin claim "alright, upon Node events, move the pod failed by me anyways". Then once this knob is registered, we move pods failed by that plugin back unconditionally; instead of proceeding with . This hook should be available to in-tree and out-of-tree plugins. In terms of in-tree plugins, it's just PodTopologySpread (we discussed this in today's sig meeting, even InterPodAffinity doesn't fall into this case) So, as a first step, let's ensure we're on the same page and agree on the ^^ summary. Next, we can better evaluate trade-offs and come up with an optimal solution.
Thanks for discussing this topic in the sig meeting. But, I am still believing that all logic related to a plugin should be controllable from the plugin side and we should not create something that can be achieved in in-tree plugins but not in out-of-tree plugins. So... the benefit of the "customizable PreCheck approach" () isn't only to solve this issue. We can use this feature for other optimization: like improving the whole performance of scheduler by moving only Pods that are more likely to be able to be scheduled by defining more detailed events on with customizable PreCheck. Many plugins have a particular scenario that makes rejected Pods schedulable and the current doesn't have an abundant enough ability of expression for this. (So, today many plugins need to say "register this hoge event on foo resource. This event may make rejected Pods schedulable" on comments in . ) Yes, I watched the sig-meeting recording as well and the opinion that the "customizable PreCheck approach" increases some implementation complexity surely makes sense. But,,, isn't this complexity increasing acceptable even if the above is taken into account?
EDIT: It is a bad habit of mine to always write long sentences even though I am not very good at English. Summarizing the above: I think it's not a good idea to create something that can be achieved in in-tree plugins but not in out-of-tree plugins. If we need a solid advantage to extend like this, then it has the advantage of optimizing the queue per plugin like described above.
I didn't jump in the thread since it's actively on the discussion. But, the issue is the similar one. (This time, we can solve that issue by changing scheduling queue logic since It's the k/k feature though.)
I'm not sure I fully understand your intention. The slack thread was discussing the AssignedPod update case, so it fallls into case 1 in , which we already agreed to improve in upstream.
Hi Are you working on this? I think this is quite important and if you're not working on this, I can follow.
Sorry, leave here for a while. I wanted to wait for (and then forgot to come back here. ) That proposal is quite similar to what I want to see in the conclusion of this issue; every plugin can judge if each Pod in unschedQ can be moved to schedQ/backoffQ. Do you think it makes sense to add "By which event the pod will be moved back to schedQ/backoffQ" on 's argument? I know and isn't needed in DefaultEnqueue plugin for API, but that's the last missing puzzle piece : If has , enqueue plugin can judge if the event makes it possible to schedule the Pod. For example, I believe we can move ↓ this logic to Pod Affinity plugin by implementing in Pod Affinity plugin. Pod Affinity plugin will remember Pods rejected by itself in . And in , if the given Pod was the one rejected by Pod Affinity, is AssignedPod/Created|Updated and is Pod affinity with the rejected Pod, then move the Pod back to activeQ/backoffQ. Otherwise reject the Pod moving. We can resolve (3) by implementing as well. (by exactly the way you said.↓) My original concern in this issue is that having logic for specific plugins in the scheduler core may prevent out-of-tree plugins' extendability. The scheduler itself should be pure, and shouldn't do anything special for any plugins implementation ideally.
If we can move all such logic to each plugin, that'd be perfect.
^ is more like a new feature than just a solution to this problem. I created another issue and describe my proposal in detail. Let's have a discussion there.
It can be resolved by moving all preCheck into each plugin's QueueingHint. Let me retitle this issue so that the current status is easy to understand. /retitle scheduler: move all preCheck to QueueingHint
This issue has not been updated in over 1 year, and should be re-triaged. You can: Confirm that this issue is still relevant with (org members only) Close this issue with For more details on the triage process, see /remove-triage accepted
/triage accepted
What happened: As part of creating an e2e test for , it was found that the apiserver proxy server is processing a blank redirect for any for 21 endpoints across pods, services and nodes. For a pod, redirecting for any method other than GET or HEAD is a bug, you expected to happen: This is the current behaviour for the apiserver but with a minimal code change the redirect can apply to only the following endpoints; - - - How to reproduce it (as minimally and precisely as possible): Create any standard pod and service in the default namespace run in another terminal the response from running the following commands produces a can be replaced with other http verbs , , , , and which also produce 301 responses. Anything else we need to know?: networking /sig testing /sig architecture
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : What happened: - - - run in another terminal the response from running the following commands produces a testing /sig architecture Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Does this issue and PR address your concerns about the apiserver proxy server doing a blanket direct of any http call to the root proxy endpoints?
commented on the associated PR, thanks
/remove-sig testing
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale Will be addressed late in 1.21
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
/remove-lifecycle stale
/remove-lifecycle stale
when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: What happened: kube-proxy came up cleanly but mis-configured iptable rules due to a mis-match in the hostname and node name within the cluster triggering a default behavior. In this case the hostname returned the fqdn of the host and the cluster only had the hostname. This resulted in the following warning messages when kube-proxy started. Which then presents itself as pods not being able to access clusterip ip addresses with no route errors. What you expected to happen: Would it be better to fail to in this scenario rather than try to fall back to some default? I spent a good amount of time trying to track down the Error messages that were a result of this before finally working back to taking a closer at the Warning message. How to reproduce it (as minimally and precisely as possible): Start the kube-proxy with set to an invalid hostname should recreate this issue. Anything else we need to know?: While I think we can chalk up this issue to mis-configuration the purpose of opening this issue is really to have a conversation about the default behavior. I've seen several other issues filed that appear to be related to this behavior: , , . Would throwing an error and exiting non-zero be a better default action when the nodeIP is nil? What use case(s) does a default of support? If there must be a default behavior here maybe the IP address on the interface associated with the default gateway is a better default than ? Maybe the issue isn't in the proxier code but the kube-proxy itself should error when it fails to get node info rather than a warning? Environment: I'm not sure how much of this is important to this but here's what my environment was when I ran into this. Kubernetes version: 1.8.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): Centos 7 and Ubuntu 16.04
This is similar (though not the same symptom) to for which was supposed to fix it but was closed. One of the objections was up til now, kube-proxy has had no requirement that the hostname = nodename or even that the hostname be defined.
Making kube-proxy exit on failure to retrieve the node object might cause regressions in some legitimate use-cases. But I have no idea what those are. ?
I have a hard time imagining what the legit cases might be. I am confused, though - I only see this field being used in one very specific place, that would not have the effect you are claiming.
I can't see the error message from log? If kube-proxy failed to create kube-ipvs0 dummy interface, it will log something like "Failed to create dummy interface: kube-ipvs0, error:...".
This breaks Services of type and with . This is problematic for AWS environments where the kubelet is configured to override the hostname (AWS instances default to the short name and for the cloud provider integrations to work this needs to be overridden to the private fqdn of the host).
Any chance we could get this prioritized as a bug fix for v1.11?
cc
After putting a bit more thought into this, would it be possible to standardize on an environment variable that could be set through the downward api to override the default behavior of treating the hostname as the node name in the proxy? That would simplify the configuration on the kubeadm side (no need to override the command with the downward api, only setting the appropriate env variable), and would also provide a more accurate way to retrieve the actual node name for kube-proxy when deployed as a daemonset without the need to interact with the kubelet directly. I would suggest that we could use for this purpose. I'd also be happy to take a stab at implementing the changes needed for the v1.12 cycle.
Pods not getting scheduled on the new node joined to the cluster using kubespray The node joined to the cluster but new pods schedueld on the nodes are failing with error "get https://10.233.0.1 : dial tcp 10.233.0.1: 443 connection refused" Network plugin - calico kube-proxy running in ipvs mode new worker node, can connect to master server ip address over 6443 new worker node, cannot access cluster ip 10.233.0.1 over port 443 , nodelocaldns and kube-proxy pods are in running state on new node -proxy logs show error "invalid nodeIP, initializing kube-proxy with 127.0.0.1 as nodeIP" Unable to write event: 'Post https://127.0.0.1:6443/api/v1/namespaces/default/events dial tcp 127.0.0.1:6443 connection refused" -ln output do not show any services addr show output do not show kube-ipvs0 and calico interfaces Can someone please help. Thanks
in W0128 10:46:06.683] [38;5;9m[FAIL][0m [0m[sig-node] Device Plugin [Feature:DevicePluginProbe][NodeFeature:DevicePluginProbe][Serial] [38;5;9m[1mDevicePlugin [Serial] [Disruptive] [BeforeEach] [0mCan schedule a pod that requires a device[0m The pod was left by the test case not sure No response No response_ /sig node
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc for (revert of )
these tests started failing after got merged?
The test passed after rebasing with the latest code. It should be fixed. Thanks.
It failed for some other remaining pods again.
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It is still an issue, see It seems that some pods are sometimes not being cleaned up after the test finishes...
Taking a look at Test fails with On kubelet side it was deleted here: The logs show that API server recorded a delete of the pod at but the next test was already running at which is before the pod was deleted. It seems like the test that created namespace (which is ), completed before it deleted all the resources created in the test... edit: The namespace was deleted here: The namespace deletion in the test framework looks async, i.e. I think it will not block to wait until the namespace is deleted... not sure if this is by design or not...
See related slack thread - Namespace deletion is async by design and the test needs to account for this. For this case in particular, since requires no pods to be left over from other tests, we can adjust the BeforeBegin to wait and poll for a bit while all the resources are cleaned up.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite} Happened on a presubmit run in .
Sorry, it is a new e2e test I am still working on. It is not merged yet.
Don't worry about it, our flake detection isn't quite perfect yet :)
[FLAKE-PING] This flaky-test issue would love to have more attention...
[FLAKE-PING] This flaky-test issue would love to have more attention.
This new e2e test has been merged. Let's observe it few more days and close this issue if nothing bad happen.
[FLAKE-PING] This flaky-test issue would love to have more attention.
This does't look like a failure in new test, rather its a generic failure with serviceaccounts.
Thanks for the investigation. Yeah I think it is the serviceaccounts issue. I wonder when could we close this flaky issue. Seems like it is stable if serviceaccounts issue is fixed.
You can close it with a reference to the serviceaccounts issue - do you know the # for that one ?
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
https://k8s- Failed: [] Services should preserve source pod IP for traffic thru service cluster IP {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Have a fix for this, I'm now keeping running tests on my own cluster. Will send out PR once verify the fix works.
When a Job is deleted (via Background cascade), we have a backup logic to remove finalizers from pods This logic could also be applied for any finished Job. While investigating a case similar to , we observed that the job controller didn't delete finalizers from Pods where the Job was deleted with Foreground cascading policy. Since the Job was already marked as Finished, it only makes sense to trigger the backup logic to remove finalizers, so that we can guard against any possible bug where a Job is marked as Finished but not all pod finalizers are removed. Note that I'm not aware of any other bug in tracking with finalizers.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig apps
/remove-kind feature /kind cleanup
/wg batch
I am happy to give this a go.
/assign
Just to check if I have the logic chain right: We are talking about adding this logic If so, I should probably turn the backup-logic code into a function that then apply that in both places? I also would be unsure of where in the unit tests this would be tested, are you able to point me to how this is tested in the deletePod function? I assume we add a pod that is never included in the "uncounted" lists.
I would put it in the event handler, to be more of a "catch all" Yes, that's a good idea. I think you would have to force it quite a bit, as we have fixed all the bugs that we have found. For example, from the test: the controller, but do not start the workers (this will setup the event handlers) a job the pods the job the workers some inspiration here
Alright, thank you :))
Should the issue title be: Remove finalizers from Pods belonging to finished Jobs ?
If someone has set a custom finalizer on a Pod, even one owned by a Job, (when) would we remove that finalizer?
/retitle Remove tracking finalizers from Pods belonging to finished Jobs No, only the finalizer specific to job tracking