jiebi/CodeConvo huggingface dataset:
Code change:
echo "Starting etcd" kube::etcd::start <ins> # Admission Controllers to invoke prior to persisting objects in cluster ADMISSION_CONTROL=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota </ins> APISERVER_LOG=/tmp/kube-apiserver.log sudo -E "${GO_OUT}/kube-apiserver"  --v=${LOG_LEVEL}  <ins> --admission_control="${ADMISSION_CONTROL}"  </ins> --address="${API_HOST}"  --port="${API_PORT}"  --runtime_config=api/v1beta3 

Related discussion:
text: Tried to create a service on a namespace 'staging' which doesn't exist on my k8s. Creation succeeded which is odd enough :) Then navigating to the entity also works: GET on /api/v1beta3/namespaces/staging/services/redis-service Navigating to the namespace itself produces 404 'namespaces "staging" not found': /api/v1beta3/namespaces/staging/ Getting all namespaces also doesn't list that namespace which sounds reasonable given the above. I'd expect one of the two: standard approach: a clear error that this namespace doesn't exist when attempting to create an entity on it. preferred imo, but better than nothing: creating silently behind the scenes that namespace cc
Sounds like an error configuring your environment. Is this using a standard provider environment or did you install your own? The kube-Apiserver needs the following flags set for admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,ResourceQuota In your case NamespaceAutoProvision must not be running. Sent from my iPhone
Admission control option NamespaceExists does your goal for 1. It is what OpenShift uses. NamespaceAutoProvision does 2. It is used upstream right now, but I think upstream should switch to NamespaceExists and require explicit creation, but that is harder to enable for things like e2e until more work is done. Sent from my iPhone
this is the hack/local-up- I didn't find the admissioncontrol set there at all, perhaps it's a bug in that particular setup?
Code change:
stepInterval  time.Duration skipIfBlocked bool destChan      chan time.Time <del> fired         bool </del> } func NewFakeClock(t time.Time) *FakeClock {

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I was writing a variant of clock.FakeClock, and noticed a couple of bugs in . Look at the pointer comparison in --- the two pointers will never be equal, but the author clearly expected equality when is the index of the fakeClockWaiter corresponding to the fakeTimer. The two pointers will never be equal because one is a pointer into the fakeClock's slice while the other pointer is to a field of the . The consequence of this bug is that the Stop function is ineffective. Also, determines the returned value based on the original copy of the waiter, which --- being a copy --- did not get updated if/when the waiter fired. Note also that does not exercise FakeClock::NewTimer at all. What you expected to happen: I expected FakeClock Timers to work properly. How to reproduce it (as minimally and precisely as possible): Use the TestFakeTimer func introduced in Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
The fake timer's Reset() method also does not work correctly.
Code change:
export KUBE_ROOT=$(dirname "${BASH_SOURCE}")/.. source "${KUBE_ROOT}/hack/lib/init.sh" <ins> git config http.https://gopkg.in.followRedirects true </ins> go get -u gopkg.in/mikedanese/gazel.v13/gazel if ! "${GOPATH}/bin/gazel" -validate -print-diff -root="$(kube::realpath ${KUBE_ROOT})" ; then echo

Related discussion:
text: When I run it locally: Small test case to induce failure: I am using git version 2.11.0, if it makes any difference.
Also applies to
This is caused by the most recent git upgrade and affects all of our deps but gazel is the one we pull the most. how do we mitigate this?
We could possibly set in verify/update-gazel, but that feels a bit iffy because it has to be global. Possibly print out the command to fix it in case the go get fails?
Source code for is here . There doesn't seem to be an open issue for this.
I filed
says
Sorry for the trouble. We'll change a bit to ensure the manual configuration is not necessary.
Was this fixed in
No need to change the configuration anymore. We've changed itself to support the new git quirks.
thanks for fixing this!
Code change:
for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

Related discussion:
text: What happened: A Pod was scheduled for deletion because a taint w/ effect was observed in the Node the Pod was assigned to, and after the taint is removed from the Node, the Pod scheduled deletion isn't canceled. After scheduling pod deletion, the only cancels pod deletion if all taints are removed from the node, including any taints that the user relies on for their use-cases (example below). What you expected to happen: After the taint that triggered scheduling pod deletion is removed, cancels the scheduled pod deletion. How to reproduce it (as minimally and precisely as possible): A node is tainted w/ , so any workloads already assigned to it which don't tolerate this taint are evicted. A pod is created w/ tolerations: w/ unspecified and w/ . is assigned to . At some point in time after the previous steps took place, is tainted as . gets an update event for and observes two taints. It proceeds to calculate the minimum time (in seconds) that tolerates for the two taints which at this point is 300 seconds, and marks for deletion in ~300 seconds - this happens in-memory, in a timed-queue. At some point in time after the previous steps took place, and before the 300 seconds trigger happen, taint is removed from . gets an update event for and observes only one taint, . It proceeds to calculate the minimum time (in seconds) that tolerates for the this taint which is infinity, and returns, not canceling the previous deletion. It completely ignores the fact that the taint that triggered deletion is no longer observed. Anything else we need to know?: Worked w/ to figure this out after all pods in a node got evicted without apparent reason. Environment: Kubernetes version (use ): 1.17.3 (but looking at the code seems to affect all of 1.17.x and 1.18.x and master. Cloud provider or hardware configuration: N/A OS (e.g: ): N/A Kernel (e.g. ): 5.x Install tools: N/A Network plugin and version (if this is a network-related bug): N/A Others: N/A cc (owners of this code)
There seems to be at least .
Code change:
for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

Related discussion:
text: You can sort of get a feel for what's going on here by looking at these two code snippets: This is what happens in when the node does not have any NoExecute taints: If you have any NoExecute taints on the node processPodOnNode gets called and this routine is really, really eager to evict pods. So : We are currently working around this issue by changing over our NoExecute taints to NoSchedule.
/sig scheduling node
/cc
/remove-sig scheduling scheduler isn't involved in this case.
Which SIG owns the controller-manager or is the node lifecycle manager (and its NoExecuteTaintManager) owned by sig-node alone?
I think node lifecycle is jointly owned by sig-node and sig-cloud-provider
API machinery owns the mechanics of the controller manager (controller loop setup and management), but the individual controllers are SIG-specific
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Looks like we lost some momentum here and I'd like to avoid this falling into the abyss of bugs that never get fixed. When this bug is triggered the outcome is pretty terrible and once it does happen it is very difficult to dig into and understand what actually happened. Did we ever get this routed to the right owner?
adding sig-node leads for triage/routing. this seems like a clearly reproducible bug with significant correctness issues. /assign
thanks for the detailed report and reproducer. to help accelerate this, do you have a form of your reproducer we could add in the e2e suite (even if its failing initially)?
custom taints are considered disruptive, so it couldn't run in the main suite
It's been a while since I wrote tests for the e2e suite but if you are available over slack in case guidance is needed, I'm willing to put the effort.
Do all suites need to pass for a fix to be merged? I'd assume it is the case, otherwise how would a regression be prevented after the fix lands?
Code change:
for _, item := range testCases { stopCh := make(chan struct{}) fakeClientset := fake.NewSimpleClientset(&v1.PodList{Items: item.pods}) <del> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </del> <ins> controller := NewNoExecuteTaintManager(fakeClientset, getPodFromClientset(fakeClientset), (&nodeHolder{node: item.node}).getNode, getPodsAssignedToNode(fakeClientset)) </ins> controller.recorder = testutil.NewFakeRecorder() go controller.Run(stopCh) controller.NodeUpdated(nil, item.node)

Related discussion:
text: i was more inquiring if there was a test that could automate the reproduction, so even just a linked commit may be helpful for someone that wants to pick this up. i am happy to help if you want to reach out over slack later this week.
looks like a unit test could reproduce the issue, opened a WIP fix in
our code is not publicly accessible, I'm sorry. PR linked by seems to cover both the fix and the test that proves it.
/remove-lifecycle frozen
frozen keeps the bot from auto-closing
Ah, sorry then.
Code change:
return nil, fmt.Errorf("%s is an incomplete jsonpath template", j.name) } <del> j.cur = []reflect.Value{reflect.ValueOf(data)} </del> <ins> cur := []reflect.Value{reflect.ValueOf(data)} </ins> nodes := j.parser.Root.Nodes fullResult := [][]reflect.Value{} for i := 0; i < len(nodes); i++ { node := nodes[i] <del> results, err := j.walk(j.cur, node) </del> <ins> results, err := j.walk(cur, node) </ins> if err != nil { return nil, err }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: (or in my instance) with does not correctly handle nested range statements. For example, one would expect that the following: would result in a list being output similar to this: but instead, results in a list being output like this: What you expected to happen: Nested range statements complete in the order in which they are specified in jsonpath query. How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.0 Cloud provider or hardware configuration: OpenShift 3.11.141 on vSphere 6.5 OS (e.g: ): RHEL 7.7 Kernel (e.g. ): 3.10.0 Install tools:
/sig cli
/area kubectl
/assign
/sig api-machinery
/remove-sig cli
/remove-area cli /area api
Those labels are not set on the issue: <detailsIn response to : /remove-area cli /area api Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-area kubectl
Code change:
}, func(client kubeclientset.Interface, obj runtime.Object) error { namespace := obj.(*apiv1.Namespace) <del> err := client.Core().Namespaces().Delete(namespace.Name, &metav1.DeleteOptions{}) </del> <ins> orphanDependents := false err := client.Core().Namespaces().Delete(namespace.Name, &metav1.DeleteOptions{OrphanDependents: &orphanDependents}) </ins> // IsNotFound error is fine since that means the object is deleted already. if errors.IsNotFound(err) { return nil

Related discussion:
text: Problem: When users sends a cascading deletion request to federation i.e they set orphanDependents=false, then federation should also set orphanDependents=false, when it sends the request to underlying clusters.
Fixed by
Code change:
Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

Related discussion:
text: In our custom plugin, we want to move pods that is unscheduled by the plugin to activeQ/backoffQ when a new Pod is created. So, we define like this: But, even when a new Pod is created, the Pod that was unscheduled by our custom plugin isn't moved to backoffQ/activeQ. when a new Pod is created, the Pod unscheduled by our custom plugin is moved to backoffQ/activeQ. a custom plugin that has pod events in EventsToRegister. a new Pod that the custom plugin will reject. another new Pod that makes the Pod created in (1) schedulable. can see the Pod created in (1) isn't scheduled immediately (it should've been scheduled immediately after (3) because (3) should move the Pod created in (1) to activeQ/backoffQ.) If I understand correctly and if it's not expected behavior, this bug has a non-small impact, especially in v1.24 because we change to 5 min. v1.23.4 but I think latest master has the same issue. <details</details<details</details<details</details<details</details<details</details
/assign /sig scheduling
Maybe it's only on a newly created unscheduled Pod. We've moved correctly for events related to assigned Pod.
And deleting Node events has the same issue...? I guess we need to run when some Nodes are deleted. In in-tree plugins, only pod topology spread uses the deleting Node event and I guess this bug affects its performance. (Even when some Nodes are deleted, the Pods, that were unscheduled by Pod topology spread, aren't moved to activeQ/backoffQ by that deleting events.)
Hm, it may be a problem that events related to Node and Pod are not always called when an event occurs. (Even if it's intended behavior, at least we need to write this on the comment of .) /retitle scheduler: events in related to Pod and Node don't always work
To summarize: The problem is "not all events related to Pod or Node don't always make move from unschedulableQ to active/backoffQ and it may not be intended for custom-plugin developers because this behavior isn't even documented." I think we should move all unschedulable Pods with all Pod and Node events.
Code change:
Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

Related discussion:
text: If concerned about performance degradation, we can extend or , for example, to have the function like ? This problem has a non-small impact, especially in v1.24+ because we change PodMaxUnschedulableQDuration to 5 min. Let's see the details... ↓ We move the unscheduled Pods (rejected by the plugin that has Pod add/update events in ) in or But, the target Pods against this moving are only unschedulable pods which have any affinity term that matches "pod". So,,,, users cannot move from unschedulableQ to activeQ/backoffQ with all assignedPod events? For example, for Pod Topology Spread (it uses Pod All event in ), the new assigned Pod creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of whether that unschedulable Pods have any affinity term that matches a newly created assigned pod or not. Of course this is not just a problem for Pod Topology Spread, it could be a problem for user's custom-plugins. I guess we should pass all unscheduled Pods to . Or at least we should document this behavior. No issue on this. We don't move the unschedulable Pods with those events at all. That's a problem... Even when a plugin uses Pod add/update/delete event in and wants to move the rejected Pods with non-scheduled Pod's add/update/delete events, scheduling queue doesn't move the unschedulable Pods with those events. (In our company, we have the custom plugin for and the plugin wants the scheduler queye to move the Pods rejected by the plugin from unschedulableQ to active/backoffQ when a new non-scheduled Pod is created.) We move only unscheduled Pods that pass . checks the Node can run that unscheduled Pod from the points of view of some in-tree filter plugins like noderesources/nodeport/nodeAffinity/nodename. The problem that arises here is similar to "Assigned Pod's add/update events". For example, for Pod Topology Spread (it uses Node Add event in ), the new Node creation may make some Pods, that are rejected by Pod Topology Spread, schedulable. Regardless of the results of against that Node. (In , we change the skew calculation of Pod Topology Spread to exclude Nodes that don't match the node affinity.
Code change:
Done(types.UID) Update(logger klog.Logger, oldPod, newPod *v1.Pod) Delete(pod *v1.Pod) <del> // TODO(sanposhiho): move all PreEnqueueCheck to Requeue and delete it from this parameter eventually. // Some PreEnqueueCheck include event filtering logic based on some in-tree plugins // and it affect badly to other plugins. // See https://github.com/kubernetes/kubernetes/issues/110175 </del> <ins> // Important Note: preCheck shouldn't include anything that depends on the in-tree plugins' logic. // (e.g., filter Pods based on added/updated Node's capacity, etc.) // We know currently some do, but we'll eventually remove them in favor of the scheduling queue hint. </ins> MoveAllToActiveOrBackoffQueue(logger klog.Logger, event framework.ClusterEvent, oldObj, newObj interface{}, preCheck PreEnqueueCheck) AssignedPodAdded(logger klog.Logger, pod *v1.Pod) AssignedPodUpdated(logger klog.Logger, oldPod, newPod *v1.Pod, event framework.ClusterEvent)

Related discussion:
text: For example, Pod Topology Spread (after is implemented). When a user uses NodeAffinityPolicy: Ignore and WhenUnsatisfiable: DoNotSchedule, the creation of Node that doesn't match nodeAffinity/nodeSelector may change the scheduling result. The skew of a new created node is always 0, so it's still unfit I think.
Our company's custom plugin is like plugin. And like coscheduling's prefilter, our plugin rejects Pods in PreFilter when the number of Pods is less than the defined number. Yes, so, maybe coscheduling plugin has the same issue as our plugin. The coscheduling plugin also uses Pod Add event and I guess it's for non-scheduled Pod's creation. (I don't know much about the coscheduling plugin though.) Oh, that's true Please forget about that example. But, how about like this scenario: user uses and Pod is rejected by Pod Topology Spread because of the skew. e.g. a user uses and TopologyKey is hostname, NodeA/NodeB has 10 Pods each, and NodeC has 8 Pods. adds un-tolerated taint to NodeC skew calculation result should be changed and the Pod rejected in (2) will be allowed by Pod Topology Spread. But the Pod isn't moved from unschedulableQ without flushing because of this issue.
Maybe we can find some pointcuts by the in , e.g. if pod was rejected by PodTopologySpread, we'll when Assigned Pod's add/update events arrived. For customize plugins, we may have to do some refactoring to register the filter func together with the event.
So now NodeC is excluded from calculations. Hence the pod fits in either NodeA or NodeB, which were unchanged. Tricky... I'll have to think about it. We really need those quick checks to pass for performance in most scenarios. /triage accepted
Thanks for bringing this up. Some comments below: You're right. Pods failed by PodTopologySpread may not get re-queued immediately. However, an assigned Pod's add/update event would only impact unschedulable Pods with cross-node scheduling directives. As of today, it's InterPodAffinity and PodTopologySpread.