jiebi/CodeConvo huggingface dataset:
Code change:
statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

Related discussion:
text: There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

Related discussion:
text: </details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.
Code change:
statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

Related discussion:
text: A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.
Code change:
statusLock            sync.Mutex running               map[types.UID]bool terminating           map[types.UID]bool <ins> terminated            map[types.UID]bool </ins> terminationRequested  map[types.UID]bool removeRuntime         map[types.UID]bool removeContent         map[types.UID]bool

Related discussion:
text: Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.
Code change:
* * If updating an existing resource: * See `Conflict` from the `status` response section below on how to retrieve more information about the nature of the conflict. * GET and compare the fields in the pre-existing object, merge changes (if still valid according to preconditions), and retry with the updated request (including `ResourceVersion`). <ins> * `410 StatusGone` * Indicates that the item is no longer available at the server and no forwarding address is known. * Suggested client recovery behavior * Do not retry. Fix the request. </ins> * `422 StatusUnprocessableEntity` * Indicates that the requested create or update operation cannot be completed due to invalid data provided as part of the request. * Suggested client recovery behavior

Related discussion:
text: We sorta fixed this-- now it's a real status error, but we left the status code 500, so it still produces stack traces and potentially confuses clients. we should figure out the correct status code.
I think there is a lot more going on related to this in . Clients are re-listing after the error but not rewatching.
/sub
It turned out this was not causing job e2e flakes. Just sending 500s that were confusing. So lowering priority.
How about a NewBadRequest error?
that certainly sounds like an improvement.
Code change:
package events import ( <ins> "context" </ins> "strconv" "testing" "time"

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I wrote a test creating multiple isomorphic Events with an "". and stumbled upon a data race. The data race happens in the function. It seems that the Event Series field can be read and written at the same time by two different goroutines. . This seems to happen in only one scenario: (1) a singleton Event is created (2) an isomorphic Event is created (1) lock addition to cache (2) wait for the lock to be released (1) unlock (2) lock + the Series (1) the freshly created Series (1) singleton Event is now considered as a series That being said, this seems to lead to some data inconsistencies as the singleton Event would be created with an unexpected Series. <details<summaryData race</summary|================== WARNING: DATA RACE Write at 0x00c0001237a0 by goroutine 23: (eventBroadcasterImpl).recordToSink.func1.1() +0x536 (eventBroadcasterImpl).recordToSink.func1() +0x84 Previous read at 0x00c0001237a0 by goroutine 24: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x56 (Event).DeepCopyObject() +0x33 () +0xe8 (CreateActionImpl).DeepCopy() <autogenerated:1 +0xb7 (Fake).Invokes() +0x103 (FakeEvents).Create() +0x214 (EventSinkImpl).Create() +0x195 () +0x32c (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).
Code change:
package events import ( <ins> "context" </ins> "strconv" "testing" "time"

Related discussion:
text: recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== |================== WARNING: DATA RACE Read at 0x00c0001236f0 by goroutine 23: (Event).DeepCopyInto() +0x64 (Event).DeepCopy() +0x72 () +0x4d () +0x98 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Previous write at 0x00c0001236f0 by goroutine 24: () +0x2e4 (eventBroadcasterImpl).attemptRecording() +0xe4 (eventBroadcasterImpl).recordToSink.func1() +0x9d Goroutine 23 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (eventBroadcasterImpl).StartEventWatcher.func2() +0x91 Goroutine 24 (running) created at: (eventBroadcasterImpl).recordToSink() +0x159 (eventBroadcasterImpl).startRecordingEvents.func1() +0x75 (*eventBroadcasterImpl).StartEventWatcher.func2() +0x91 |================== </detailsWhat you expected to happen: No data race. How to reproduce it (as minimally and precisely as possible): You can run the following code snippet alongside any data race detector. I personally used . Anything else we need to know?: Environment: v0.25.0
/sig api-machinery
/assign Would you mind taking a look to this one? thanks!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity.
Code change:
package events import ( <ins> "context" </ins> "strconv" "testing" "time"

Related discussion:
text: If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle rotten
/remove-lifecycle rotten
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
Stale issues rot after 30d of inactivity. Mark the issue as fresh with . Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle rotten
Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . Send feedback to sig-contributor-experience at . /close
Closing this issue. <detailsIn response to : Rotten issues close after 30d of inactivity. Reopen the issue with . Mark the issue as fresh with . /close Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We might also want to look into that again for /reopen /assign /unassign
/remove-lifecycle rotten
Reopened this issue. <detailsIn response to : We might also want to look into that again for /unassign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/cc
/cc
I was able to reproduce this issue with client-go v0.25.0. I updated the reproducer to be compatible with the latest version of the client and attached the data race trace.
are you gong to submit a fix for this one? thanks. /triage accepted
Yes, I am planning to submit a fix for this issue.
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.
Code change:
package events import ( <ins> "context" </ins> "strconv" "testing" "time"

Related discussion:
text: This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
A PR is out to fix this issue: /remove-lifecycle stale
Code change:
printFlags := GoTemplatePrintFlags{ TemplateArgument: templateArg, } <ins> if !sort.StringsAreSorted(printFlags.AllowedFormats()) { t.Fatalf("allowed formats are not sorted") } </ins> p, err := printFlags.ToPrinter(tc.outputFormat) if tc.expectNoMatch {

Related discussion:
text: /kind bug What happened: When building kubernetes-1.7.0 packages for openSUSE, results differ for each build in and other man pages. coming from that uses and probably sets do not have any specific order, so it would need to be sorted on output. What you expected to happen: Every build should give identical results. See https://reproducible- for why this matters. How to reproduce it (as minimally and precisely as possible): See Anything else we need to know?: Environment: Kubernetes version: 1.7.0 OS (e.g. from /etc/os-release): openSUSE Tumbleweed
/sig release
somehow this bug seems to have been forgotten - can you help getting the right people to have a look?
- can you please confirm that PR works for you?
I tested it and it looks good.
/assign I enjoy reading my highlights to find the issue has already been addressed. :)
Sorry this wasn't addressed sooner. It does seem like it fell into a black hole, and pinging was the right way to resolve that. :)
/area build-release
related to /reopen
Reopening this issue. <detailsIn response to : related to Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
"github.com/stretchr/testify/assert" ) <del> func TestGetAppArmorFS(t *testing.T) { // This test only passes on systems running AppArmor with the default configuration. // The test should be manually run if modifying the getAppArmorFS function. t.Skip() const expectedPath = "/sys/kernel/security/apparmor" actualPath, err := getAppArmorFS() assert.NoError(t, err) assert.Equal(t, expectedPath, actualPath) } func TestValidateHost(t *testing.T) { // This test only passes on systems running AppArmor with the default configuration. // The test should be manually run if modifying the getAppArmorFS function. t.Skip() assert.NoError(t, validateHost()) } </del> func TestValidateBadHost(t *testing.T) { hostErr := errors.New("expected host error") v := &validator{

Related discussion:
text: As mentioned the return value of is not being used any longer and the entire method can be removed. The usage of was removed in Now there is no need to have the function and store it's result at all. /sig node /kind cleanup /good-first-issue /help-wanted KEP xref:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : As mentioned the return value of is not being used any longer and the entire method can be removed. /kind cleanup /good-first-issue /help-wanted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/triage accepted
/assign
I am having issues submitting the pull request. I removed that portion of the code under however this is my first time contributing and would like some help submitting this new updated version..
/assign
/assign
/assign
/assign
Code change:
} c.invalidateCachedPredicatesOnNodeUpdate(newNode, oldNode) <del> c.podQueue.MoveAllToActiveQueue() </del> <ins> // Only activate unschedulable pods if the node became more schedulable. // We skip the node property comparison when there is no unschedulable pods in the queue // to save processing cycles. We still trigger a move to active queue to cover the case // that a pod being processed by the scheduler is determined unschedulable. We want this // pod to be reevaluated when a change in the cluster happens. if c.podQueue.NumUnschedulablePods() == 0 || nodeSchedulingPropertiesChanged(newNode, oldNode) { c.podQueue.MoveAllToActiveQueue() } </ins> } func (c *configFactory) invalidateCachedPredicatesOnNodeUpdate(newNode *v1.Node, oldNode *v1.Node) {

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : Today, the scheduler re-processes unschedulable nodes . Node updates are sent by Kubelet frequently (every 10 seconds) and often times they don't change any of the node properties that could potentially make them feasible. We should be smarter about these updates and move pods to the schedulable queue only when certain properties of nodes change that could make them feasible. These properties include node allocatable, node conditions, taints, labels, .... Why is this needed: Improves performance of the scheduler and avoids doing unnecessary attempts to schedule pods. This is particularly helpful when a cluster has a lot of unschedulable pods. <!-- DO NOT EDIT BELOW THIS LINE --/kind feature This is not a feature really. It is performance optimization, but in the absence of such a label, I need to use "feature" here. /sig scheduling
I can help to fix it. /assign mlmhl
The main reason for this problem is / update their status too frequently and most of these updates are heartbeats without any meaningful change. Recently sig-node introduces a new to alleviate this problem. With this mechanism, kubelet will split heartbeat and node status updates into two independent operations, and reduce the frequency of status updates to once per 1 minute(or more). In the long run this mechanism can indeed alleviate our problems, but at present it does not help us much as this is still an alpha feature and the node status updates frequency is still a bit high(per 1 minute right now).
Maybe we can try comparing old and new nodes after stripping their heartbeat info, like this:
I think that had sent earlier is more efficient. However, that PR does not filter out . In Wei's code, the heartbeat time is removed correctly. I think we should fix this issue and resubmit that PR.
could you resubmit a PR to: disregard in conditions comparison: and (optionally) disregard timestamp in taints comparison:
Thanks for all your advice, a new PR is submitted: This PR does not include your second suggestion(disregard timestamp in taints comparison) as I'm not sure if can be changed when node updating.
Code change:
} c.invalidateCachedPredicatesOnNodeUpdate(newNode, oldNode) <del> c.podQueue.MoveAllToActiveQueue() </del> <ins> // Only activate unschedulable pods if the node became more schedulable. // We skip the node property comparison when there is no unschedulable pods in the queue // to save processing cycles. We still trigger a move to active queue to cover the case // that a pod being processed by the scheduler is determined unschedulable. We want this // pod to be reevaluated when a change in the cluster happens. if c.podQueue.NumUnschedulablePods() == 0 || nodeSchedulingPropertiesChanged(newNode, oldNode) { c.podQueue.MoveAllToActiveQueue() } </ins> } func (c *configFactory) invalidateCachedPredicatesOnNodeUpdate(newNode *v1.Node, oldNode *v1.Node) {

Related discussion:
text: If it can be changed, I will add this filter later, otherwise I think we can keep the current implementation to make the compare operation simpler and faster. What's more, just as I said in the PR's description, I'm not sure how much side effects this change has on scheduler's resource consumption, as it always copy node's conditions before comparison.
I am thinking whether we could move this logic(whether node became more schedulable) to kubelet, kubelet reports it through heartbeats.