jiebi/CodeConvo huggingface dataset:
Code change:
cache.moveNodeInfoToHead(node.Name) } <del> cache.nodeTree.RemoveNode(node) </del> <ins> if err := cache.nodeTree.RemoveNode(node); err != nil { return err } </ins> cache.removeNodeImageStates(node) return nil }

Related discussion:
text: <!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) Increase the log level of info logs to "v=2" when the scheduler can't schedule a pod , , and add a log message for 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /sig scheduling /priority important-soon /assign
GitHub didn't allow me to assign the following users: congliu01. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : <!-- Please only use this template for submitting enhancement requests --1) Increase the log level of info logs to "v=2" when the scheduler or a node from its cache. 2) report memory/cpu status of . Why is this needed: With cluster autoscaler now being used in production by some vendors, we want to have more visibility to events that could be related to the auto scaler to help debug customer issues. /priority important-soon /assign </details
will consider this assigned to you.
/assign
Code change:
import ( "bytes" <del> "encoding/json" "fmt" </del> "io/ioutil" <del> "net/http" </del> "os" "path/filepath" "regexp" <del> "strings" "sync/atomic" </del> "testing" <del> "time" </del> "k8s.io/api/core/v1" "k8s.io/apimachinery/pkg/api/errors"

Related discussion:
text: Just got this while running the tests:
let's please move this to /close /sig cluster-lifecycle
Code change:
// GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

Related discussion:
text: There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
// GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

Related discussion:
text: </details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.
Code change:
// GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

Related discussion:
text: A pod the pod worker hasn’t seen yet is also filtered out (that means bursts of pods can get in, which seems the opposite of this problem, and is the bug i mentioned before).
Edit: one wrong thing spotted while looking - I'm not sure it fixes the issue you're seeing, but it's definitely wrong for admission (admission is filtering the known pod set down to remove pods that are terminated, but shouldn't filter pods that aren't admitted yet). This needs a node e2e reproducer that can run in serial (resource consumption is really hard to test from parallel, is slow in serial, and node e2e is back in a good state now) - / is that something either of you could help me with? I believe the reason the existing tests don't catch this is the job controller is creating pods fast enough that we see races in admission (multiple pods are admitted in one call to HandlePodAdditions) that are not caught in the current serial node e2e admission tests which rarely create more than one at a time.
Job pods still go through the scheduler to get assigned to a node, so even though the job controller can create pods in parallel, I'd still expect them to arrive at the node in an orderly fashion. It's still possible multiple arrive in a single HandlePodAdditions call, of course. It seems like kubelet admission is using laggy information that doesn't reflect pod completion the kubelet itself wrote to the pod API.
The bug turned out to be much simpler. Pods that are completed never go to the pod worker, so once a pod has reached terminal state it was being back to the list of active pods. The set of e2es that would have prevented this regression: node e2e that creates multiple pods at the same time and verifies that admission works correctly when HandlePodAdditions takes two pods at once node e2e that verifies that completed pods don't consume resources for admission purposes in the kubelet job controller e2e that attempts to verify some part of this reliably (ideally in a way that can be run in parallel e2e)
another related issue:
Linking here for completeness - a similar symptom might get exposed after this fix as described in The kubelet considers resources for terminating pods to be in use (they are!), but the scheduler ignores terminating pods and schedules new pods.
Code change:
// GetActivePods returns pods that may have a running container (a // terminated pod is one that is known to have no running containers and // will not get any more). <ins> // // TODO: This method must include pods that have been force deleted from // the config source (and thus removed from the pod manager) but are still // terminating. </ins> func (kl *Kubelet) GetActivePods() []*v1.Pod { allPods := kl.podManager.GetPods() activePods := kl.filterOutTerminatedPods(allPods)

Related discussion:
text: Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.
Code change:
// Visit implements Visitor over a stream. func (v *StreamVisitor) Visit(fn VisitorFunc) error { <del> d := json.NewDecoder(v.Reader) </del> <ins> d := yaml.NewYAMLOrJSONDecoder(v.Reader, 4096) </ins> for { ext := runtime.RawExtension{} if err := d.Decode(&ext); err != nil {

Related discussion:
text: If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)
Code change:
"context" "errors" "fmt" <del> "net" </del> "net/url" "strings" "time"

Related discussion:
text: Tests in are using image for gRPC probe tests: should not be restarted with a GRPC liveness probe should be restarted with a GRPC liveness probe We should use image for these tests as it is much smaller size. /sig node /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue KEP:
This request has been marked as suitable for new contributors. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : Tests in are using image for gRPC probe tests: /kind cleanup /priority important-soon /area tests /help-wanted /good-first-issue Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Hi can I work on this issue? I'm new to Kubernetes and want to contribute!
Should I change to in this file?
Yes.
Code change:
// are found. workqueue.ParallelizeUntil(ctx, 16, allNodes, checkNode) processedNodes := int(filteredLen) + len(filteredNodesStatuses) + len(failedPredicateMap) <del> g.lastProcessedNodeIndex = (g.lastProcessedNodeIndex + processedNodes) % allNodes </del> <ins> g.nextStartNodeIndex = (g.nextStartNodeIndex + processedNodes) % allNodes </ins> filtered = filtered[:filteredLen] if err := errCh.ReceiveError(); err != nil {

Related discussion:
text: What would you like to be : A unit test that verifies the evaluation order of the nodes in the scheduler. Why is this needed: The scheduler evaluates the nodes in a specific order until it finds a pre-defined minimum number of feasible nodes. To give all nodes a fair chance of being evaluated, in each scheduling cycle, the scheduler starts evaluating from where it left off in the previous cycle. This is captured by the parameter in  We need to add a unit test that captures regressions related to this behavior to avoid bugs like /sig scheduling /priority important-soon /help
/assign
/remove-help
Code change:
if activeQueues == 0 { return 0 } <del> return math.Min(float64(reqs), float64(qs.dCfg.ConcurrencyLimit)) / float64(activeQueues) </del> <ins> return math.Min(float64(seatsInUse), float64(qs.dCfg.ConcurrencyLimit)) / float64(activeQueues) </ins> } // timeoutOldRequestsAndRejectOrEnqueueLocked encapsulates the logic required

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --I noticed that the the code in for calculating the dR/dt, the derivative of R(t) with respect to t, called in the code , was not updated when the generalization for width was introduced. The should be between C and the sum of seats not the number of requests. See , where it says Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
yes, i noticed it, I am going to open a PR. /assign
/triage accepted
Code change:
volumeMounts: - name: www-data mountPath: /srv/www <del> readOnly </del> <ins> readOnly: true </ins> containers: - name: git-monitor image: kubernetes/git-monitor

Related discussion:
text: Which jobs are failing: Since when has it been failing: Testgrid link: Reason for failure: Anything else we need to know: /priority important-soon /cc /cc /milestone v1.18 /sig release
/assign
This is failing because a 1.18 release for CRI tools doesn't exist yet. I can tweak the test job arguments to skip this later.
(As a note, this job is important, but not ) /remove-priority critical-urgent /priority important-soon
Do we need to remove the job from dashboard?
-- No. I'll work on fixing the job later today.
I've started to investigate, but I haven't yet reproduced the new error locally: This is likely related to a recent refactor of the package that handles interpreting deb/rpm package versions. As is not currently in use, I'm moving this to the milestone. /milestone v1.19
the version issue is fixed via The test is now failing because it tries to download cri-tools v1.18.0: which I think is related to this code part:
Updating testgrid link here: Any updates here /retitle [Flaky Test] build-packages-debs (ci-release-build-packages-debs) /kind flaky-test
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : Updating testgrid link here: /kind flaky-test Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/kind flake
I was assuming that is working on this issue. I can jump in if you want and do not find the time to work on it. :)
/assign /unassign Sascha -- Can you take this? I thought I was going to have time to look at this, but it's been a while.
Sure, the reason for the failing test seems to have changed, I'm on it.
Fix is incoming in
Code change:
ResourceName string Command      []string <del> FullCmdName       string SuggestedCmdUsage string </del> <ins> ParentCommandName       string EnableSuggestedCmdUsage bool </ins> Builder          func() *resource.Builder ExecutablePodFn  polymorphichelpers.AttachablePodForObjectFunc

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: The pod name is missing in suggested cmd usage when executing : What you expected to happen: The suggested cmd usage should be a complete cmd like the one got from How to reproduce it (as minimally and precisely as possible): Use to attach any pod/deployment/service, the PodName will be missing in the suggested cmd usage like above. Anything else we need to know?: It works fine for kubectl < v1.12 Environment: Kubernetes version (use ): v1.13.1 Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig cli /assign
Code change:
opts.Tail = strconv.FormatInt(*logOptions.TailLines, 10) } <ins> if logOptions.LimitBytes != nil { // stdout and stderr share the total write limit max := *logOptions.LimitBytes stderr = sharedLimitWriter(stderr, &max) stdout = sharedLimitWriter(stdout, &max) } </ins> sopts := libdocker.StreamOptions{ OutputStream: stdout, ErrorStream:  stderr, RawTerminal:  container.Config.Tty, } <del> return d.client.Logs(containerID.ID, opts, sopts) </del> <ins> err = d.client.Logs(containerID.ID, opts, sopts) if errors.Is(err, errMaximumWrite) { klog.V(2).Infof("finished logs, hit byte limit %d", *logOptions.LimitBytes) err = nil } return err </ins> } // GetContainerLogTail attempts to read up to MaxContainerTerminationMessageLogLength

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Conformance test failed: xref What you expected to happen: I expected this conformance test to pass. How to reproduce it (as minimally and precisely as possible): We have configured docker's logging driver to use journald, and then ran sonobouy tests against a k8s 1.16.3 cluster. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: OS (e.g: ): Red Hat Enterprise Linux Server release 7.7 (Maipo) Kernel (e.g. ): 5.3.11- SMP Tue Nov 12 14:45:59 EST 2019 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
There are no sig labels on this issue. Please add a sig label by either: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. See the . The in method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
/sig cli
/assign
I have a working patch that fixes this, will push a PR shortly.
Code change:
go_library( name = "go_default_library", <del> srcs = ["logs.go"], </del> <ins> srcs = [ "logs.go", "options.go", "registry.go", ], </ins> importmap = "k8s.io/kubernetes/vendor/k8s.io/component-base/logs", importpath = "k8s.io/component-base/logs", deps = [ "//staging/src/k8s.io/apimachinery/pkg/util/wait:go_default_library", <ins> "//vendor/github.com/go-logr/logr:go_default_library", </ins> "//vendor/github.com/spf13/pflag:go_default_library", "//vendor/k8s.io/klog/v2:go_default_library", ],

Related discussion:
text: What would you like to be : As part of should add logging configuration allowing to change logging implementation. Things to implement: --logging-format - flag to pick between available implementations. Defaults to "text" using unchanged klog. LoggingConfig - structure storing component configuration for logging LogFormatRegistry - Used for registration of additional log formats. New formats should be registered by providing unique name and implementation of interface LoggingConfig.Complete method should configure klog by passing logr interface implementation into klog.SetLogger method Why is this needed: Subtask of /help /wg component-standard
The klog 1.0.0 version looks like no SetLogger method, should update klog version in vendor?
yes, forgot about that. I remember saying that it will require some changes, but shouldn't be to hard. I will help with it if needed.
/assign /remove-help
Klog was upgraded to v2.