jiebi/CodeConvo huggingface dataset:
Code change:
import ( "context" "fmt" <del> "reflect" </del> "strings" "time" v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" <ins> "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/runtime/schema" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.
Code change:
import ( "context" "fmt" <del> "reflect" </del> "strings" "time" v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" <ins> "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/runtime/schema" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.
Code change:
import ( "context" "fmt" <del> "reflect" </del> "strings" "time" v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" <ins> "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/runtime/schema" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.
Code change:
import ( "context" "fmt" <del> "reflect" </del> "strings" "time" v1 "k8s.io/api/core/v1" storagev1 "k8s.io/api/storage/v1" <ins> "k8s.io/apimachinery/pkg/api/equality" </ins> "k8s.io/apimachinery/pkg/runtime/schema" utilruntime "k8s.io/apimachinery/pkg/util/runtime" "k8s.io/apimachinery/pkg/util/wait"

Related discussion:
text: Let me summarize each: Born from from is right. But, that's the problem with preCheck. (Some preCheck depends on some in-tree plugins' logic and it affects badly to other plugins enqueueing ) The preCheck is removed after is completed. After preCheck is removed, will be the correct behavior. So, we can do the optimization that changes node to a specific event. will be fixed in: It's kind of rare case, but I feel we need to have some improvement like Edit: create another issue for this: I'd say (2) and (3) are kind of rare cases. So, the blocker here is (1). (And, (1) is blocked by the removal of preCheck, which is blocked by ) As my conclusion, it's safer to add after (1) is done so that we can make sure the performance won't be affected badly. what do you think?
Instead, can we just keep an event for "node updated", then individual plugins can decide whether a annotations change is worth requeuing or not?
Yes, (1) can be replaced with that idea. But, I'd say changing to a specific is slightly better because the scheduling queue could filter out some Node update events without executing the hint functions.
If rescheduling can be triggered with precise events, that would be great. I haven't revisit since last time, I'll take a look again.
Since we are missing an event for , can we add it? But we also have to modify existing events like so: Because change of such field is also an Update. And in future releases we can get rid of some fine-grained events.
Since we will add hintFuncs for all in-tree plugins, so I think it's ok to add nodeAnnotationsChanged event now, we can filter out unrelated events in hitFuncs. Any other concerns? cc Prefer to keep them as people can register fine-gained events instead a general event.
If by "now" you mean v1.30, then I'm ok with that.
Code change:
template: &api.PodTemplateSpec{ Spec: api.PodSpec{ TopologySpreadConstraints: []api.TopologySpreadConstraint{ <del> {TopologyKey: `foo`}, {TopologyKey: `beta.kubernetes.io/arch`}, {TopologyKey: `beta.kubernetes.io/os`}, {TopologyKey: `failure-domain.beta.kubernetes.io/region`}, {TopologyKey: `failure-domain.beta.kubernetes.io/zone`}, {TopologyKey: `beta.kubernetes.io/instance-type`}, </del> <ins> { TopologyKey:   `foo`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/arch`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/os`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `failure-domain.beta.kubernetes.io/region`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `failure-domain.beta.kubernetes.io/zone`, LabelSelector: &metav1.LabelSelector{}, }, { TopologyKey:   `beta.kubernetes.io/instance-type`, LabelSelector: &metav1.LabelSelector{}, }, </ins> }, }, },

Related discussion:
text: Currently, when the Label Selector in the Pod Affinity is empty, no objects will be match. That means if it's in a required PodAffinity, the Pod will never be schedulable. And if it's in a required PodAntiAffinity, that PodAntiAffinity will be no-op. So, I'm thinking PodAffinity with empty Label Selector is very likely mistake. It'd be good if we could notice it for users. To prevent such a misconfiguration. Probably, we cannot filter out such a Pod creation in the validation for a backward compatibility. /assign /sig scheduling
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I'm wondering if it has already discussed somewhere or not. Please just let me know if we have some decisions on it.
I am not sure, where do you propose to put the warning? If the pod is unschedulable, that will show up in the events and pod status.
Do you mean the place to implement this warning? I suspect ↓ is a suitable place:
A warning sounds useful, even if the pod is unschedulable, as it gives you a heads up at the time when you create the Pod. Whereas the Pod scheduling failure might be a bit harder to interpret.
ok, I thought you were referring to something that the scheduler will produce after attempting to schedule the pod. Yeah, if there is already a setup to print warnings on pod creation, it is certainly useful for this case.
Can you please also add it in topology spread?
Sure
Code change:
import ( "fmt" "reflect" <ins> "strings" </ins> "testing" v1 "k8s.io/api/core/v1"

Related discussion:
text: There is no reason this should be allowed to happen, especially that we can't configure duplicate plugins differently since plugin configs is a separate list . /sig scheduling
/label help-wanted
The label(s) cannot be applied. These labels are supported: <detailsIn response to : /label help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/help-wanted
/help /good-first-issue
/assign
/remove-good-first-issue /remove-help
Code change:
"custom": resource.MustParse("0"), }, Limits: v1.ResourceList{ <del> "memory": resource.MustParse("2G"), </del> <ins> "memory": resource.MustParse("2.5Gi"), </ins> "custom": resource.MustParse("6"), }, }},

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I found that for with returns a very small value. The value should be almost the same as the one of . My minimal : Result: While the top three lines () are good, the float values in the bottom three lines () might be broken. Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig api-machinery
/assign
/cc /triage accepted
Code change:
t.Fatalf("unexpected error: %v", err) } if !reflect.DeepEqual(testCase.Limiter, client.rateLimiter) { <del> t.Fatalf("unexpected rate limiter: %#v", client.rateLimiter) </del> <ins> t.Fatalf("unexpected rate limiter: %#v, expected %#v at %s", client.rateLimiter, testCase.Limiter, testCase.Name) </ins> } }) t.Run("Unversioned_"+testCase.Name, func(t *testing.T) {

Related discussion:
text: Reading TestRESTClientLimiter in I noticed a few things. test case struct is defined to have a name field, and its value is used in naming the subtests, but the actual test cases leave the name blank. error statements do not identify which case is failing (do not identify the expected value). Should be clear from the above. Read the code. No response_ <detailsbranch </details<detailsN/A </details<detailsN/A </details<details</details<details</details<details</details
/sig api-machinery
/assign
/triage accepted
Code change:
inf "gopkg.in/inf.v0" ) <ins> var ( bigMostPositive = big.NewInt(mostPositive) bigMostNegative = big.NewInt(mostNegative) ) </ins> func dec(i int64, exponent int) infDecAmount { // See the below test-- scale is the negative of an exponent. return infDecAmount{inf.NewDec(i, inf.Scale(-exponent))} } <ins> func bigDec(i *big.Int, exponent int) infDecAmount { // See the below test-- scale is the negative of an exponent. return infDecAmount{inf.NewDecBig(i, inf.Scale(-exponent))} } </ins> func decQuantity(i int64, exponent int, format Format) Quantity { return Quantity{d: dec(i, exponent), Format: format} } <ins> func bigDecQuantity(i *big.Int, exponent int, format Format) Quantity { return Quantity{d: bigDec(i, exponent), Format: format} } </ins> func intQuantity(i int64, exponent Scale, format Format) Quantity { return Quantity{i: int64Amount{value: i, scale: exponent}, Format: format} }

Related discussion:
text: I'd like to add a multiplication method for the resource.Quantity. Currently, the resource package provides helper methods to calculate Quantity, such as and : Batch workloads often replicate the same PodTemplate, and controllers calculate the resources needed for all containers. In that case, It is inconvenient to call the function as many times as podTemplate, so I would like to create a helper function for multiplication.
/cc /sig api-machinary /wg batch
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /cc /sig api-machinary /wg batch </details
/sig api-machinery /wg batch
Do you have an example of usage for these utilities? Just curious?
/triage accepted
Sorry for the late response. Taking k-sigs/kueue as an example, I assumed the below: It helps the controllers for batch workloads such as k-sigs/kueue, k-sigs/jobset, kubeflow/mpi-operator, and kubeflow/training-operator. I'm thinking of implementing this function once the code freeze term is finished.
/assign
Hi, may i take this up?
I took this one, and working on this now.
Code change:
NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

Related discussion:
text: Due to limited character space, I've stored additional information in annotations on nodes. My custom scheduling plugins read these annotations to schedule Pods. However, when a Pod fails to be scheduled on a node, I want it to be removed from the Unschedulable queue. This can be achieved by modifying the annotations of the node, allowing the Pod to be rescheduled. Unfortunately, the eventHandler of the scheduler only supports five types of node events, and it does not include the nodeAnnotationsChanged event. Consequently, I am currently unable to trigger rescheduling by changing the annotations of nodes. /sig scheduling Unschedulable pods will be rescheduled after annotations of nodes changed. None No response <details</details<details</details<details</details<details</details<details</details<details</details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/assign
It's more like a new feature, not bug because it doesn't lead to any bad behavior on the upstream. /remove-kind bug /kind feature
My concern is node annotation is arbitrary, change a node annotation doesn't imply a pod is schedulable. But for other events, like node labels, node taints, they're the constraints we consider in scheduling. They're different, I'm afraid this will lead to invalid scheduling attempts. Who is responsible for change the annotations in you scenario?
This enhancement has no impact on the in-tree plugins / upstream scheduler, right? Only (custom) plugins, that want to reschedule Pods via the node annotation change, will register event to EventsToRegister.
I have a controller capable of modifying node annotations to indicate node's current availability. And my custom plugins need to be aware of these changes.
I guess we can still be impacted when scheduling pod failed because of no fit node rather than failed by the plugin.
Code change:
NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

Related discussion:
text: see Can we predict whether the pod is schedulable before running into scheduling cycle via the node annotations, then we can leverage the preEnqueue extension point since you already have a external controller.
Thanks for the advice. But I'm afraid it would not be the best solution for us, because doing that outside the scheduling cycle would cause the custom plugins to malfunction in my scenario.
The concerns raised by make sense to me. In my understanding, most in-tree plugins will classify the event as an event.
No. When no fit Pod, UnschedulablePlugins will have plugin(s) rejected the Pod. (e.g., all nodes don't have enough capacity, UnschedulablePlugins has noderesourcefit. all nodes don't satisfy node affinity, UnschedulablePlugins has nodeaffinity) is a rare case like plugins other than PreFilter/Filter returns non-success. I don't think we should care about it.
But, this is a good point, I think we should refactor all plugins that register event with to have a specific Node update event. For example, . It should have with instead of . event with , that should be the last-resort when we don't know which kind of Node update will make Pods schedulable. (I guess no such case at least in the in-tree plugins.)
1) One possibility is Reserve Plugin, I the capability here 2) Another possibility is PreFilter, but not because failed by the plugin, but no fit node exists Have no idea whether we have other places, I guess no. But yes, this only impacts unschedulable pods has no failed plugins.
~I think this is for intended, like ~ ~1. 1) we have a plugin failed by nodeAffinity~ ~2. 2) we add a new node~ ~3. 3) we receive the node add event, but because node not ready, we failed by nodeTaint plugin as ~ 4) node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event)~ ~5. 5) pod will not be moved to the active or backoff queue~ updated below
weigh in for advices as whether this is a good idea to take nodeAnnotations as a new event.
Code change:
NodeAllocatableChange, NodeConditionChange, NodeLabelChange, <ins> NodeAnnotationChange, </ins> PvcAdd, PvcUpdate, PvAdd,

Related discussion:
text: As I know, some plugins will read the embedded prometheus metrics in node annotations for scoring, but not for filtering. At least we should mitigate the impacts.
iv. node become ready, we get the node update event, but dismatch with nodeAffinity registered event (if we registered with UpdateNodeLabel event) (3) is not exact. When we receive the node add event, the scheduler retry to schedule the Pod because the nodeaffinity has Node add event. And it's rejected by the nodetaint plugin. (not node affinity) So, in (4), we get the node update event, it matches with node taint registered event, and the Pod is moved to active/backoffQ (node taint registers node update event. Yes I think we can change node taint's node update event to more specific one(s) though)
As I replied in your PR, we acknowledged that in , and yes let's have your patch. Well, that's a good point. So, when all nodes are filtered out by PreFilterResult, then no failedplugin can be returned. That's a problem. I think we should return the plugin name that returns PreFilterResult with not all Node names in this case (= the plugins that contributes to reducing the candidate Nodes via PreFilterResult). What do you think?
Sorry, the previous one is not right, a detailed one: a. we have a pod failed by nodeAffinity. pod will be placed into the unschedulablePods b. node autoscaler triggered, a new node , emitted a node add event c. called -d. node not ready right now and holds a notReady taint, then failed, pod still stays in the unschedulablePods e. node gets ready, emitted a node update event, but dismatch with the registered event(if we register with in nodeAffinity plugin) f. pod will stay in unschedulabelPods I guess this is the right flow. Anyway, glad to see it be more effect in the future.
If we have no performance problems, I think it's ok to take nodeAnnotations into consideration.
Wouldn't this be solved with proposal on ClusterEventWithHint? If so, I prefer we leave the existing code as-is.
whatever we do here, we wouldn't backport, so I think is the right move forward.
We put many stuff here.