jiebi/CodeConvo huggingface dataset:
Code change:
ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

Related discussion:
text: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue or PR with Mark this issue or PR as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen being fixed by
Reopened this issue. <detailsIn response to : /reopen being fixed by Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
Code change:
ginkgo.By("scheduling a pod with a container that verifies %COMPUTERNAME% matches selected node name") image := imageutils.GetConfig(imageutils.BusyBox) podName := "host-process-test-pod" <ins> // We're passing this to powershell.exe -Command. Inside, we must use apostrophes for strings. command := fmt.Sprintf(`& {if ('%s' -ine "$(hostname)") { exit -1 }}`, targetNode.Name) </ins> pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ Name: podName,

Related discussion:
text: This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
proper information. release: v1.9 file: test/e2e/kubectl/kubectl.go <ins> - testname: Kubectl, diff Deployment codename: '[sig-cli] Kubectl client Kubectl diff should check if kubectl diff finds a difference for Deployments [Conformance]' description: Create a Deployment with httpd image. Declare the same Deployment with a different image, busybox. Diff of live Deployment with declared Deployment MUST include the difference between live and declared image. release: v1.19 file: test/e2e/kubectl/kubectl.go </ins> - testname: Kubectl, create service, replication controller codename: '[sig-cli] Kubectl client Kubectl expose should create services for rc  [Conformance]' description: Create a Pod running agnhost listening to port 6379. Using kubectl

Related discussion:
text: and server-side dry-run are being promoted to stable features in 1.18. Conformance should ensure that cloud providers support these features. As a user, I expect to work, which requires that cloud providers for cloud default webhooks. Conformance would generally avoid cloud-specific issues like this one: This suggestion is from /area conformance
/sig testing /sig architecture
/wg apply
There are some good references for conformance tests in the conformance meeting notes:
once merges, we need to promote the e2e test to conformance in a follow up PR, in accordance to this doc: It seems like we meet the conformance test requirements. Here's the process specifically, which includes sig-arch and a review board group:
Code change:
return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

Related discussion:
text: There appears to be a change of behavior / possible regression in k8s 1.22 with respect to pod lifecycle behavior. We've started to see some cases when multiple pods are being created (often as part of a k8s job) the following will happen: pods get created by e.g. job controller (e.g. say 5). Each pod in job requests majority of node memory so only one such pod can fit on node at a time. schedules [first] pod to node, rest (4 pods) stay in pending accepts first pod, runs it to completion pod get scheduled to the node rejects the pod during pod admission, e.g. with , despite the fact that the first pod has already completed successfully This results in many of pods backing the job getting into Failed phase and having to be recreated by job controller. The job can often get into BackofLimit exceeded to hitting backoffLimit. The root issue seems to be a mismatch of when kubelet actually considers the pod to complete and what it reports to API server (which causes the followup pods to be scheduled to it). I'm still investigating further, but this may be related to the pod lifecycle changes in 1.22 -- I've created a e2e test as minimal repro of this behavior -- (Test fails) 22 build log -- 22 kubelet log -- (Test passes) 21 build log -- 21 kubelet log -- <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --- Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository.
Code change:
return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

Related discussion:
text: </details
/sig node
/cc
Looking at the kubelet log in 1.22 () here's what I see -- First pod starts up... starts dies after ~10 seconds since the job is completed (the container simply runs a ) Pod completes, status is updated to API server Next pod comes in but is rejected with despite the earlier pod already being completed/terminated
did the kubelet pod lifecycle rework result in a reordering of kubelet resource bookeeping w.r.t. pod status? it seems reasonable for the scheduler to send another pod to a kubelet once it reports the pod complete and the use of the resources is done from an API perspective
I am also seeing this same behavior, but with "OutOfPods" rather than "OutOfMemory", when running a job with parallelism=100, completions=400 on GKE (which has a 110 pod per node limit).
this is easily reproducible on a local cluster as well start with this: then create this job: result on 1.22 is the job hangs with 2 completions and a bunch of rejections by the kubelet: on 1.21, the same job completes all pods successfully in a couple minutes with no OutOfpods errors
verified introduced the OutOfPods issue
Shame we didn’t have an e2e which caught this given how easy the reproducer is. This looks to be the opposite of the previous cpu admission bug (ignoring things that are terminating but not terminated). We are weak on admission testing in general. This potentially implies admission is looking at the wrong set of pods (admitted + terminating). Note there is already a bug in 1.21 and before where force deleted pods are incorrectly ignored from the set of allocated resources while they are terminating, but this doesn’t immediately seem to be that issue (since i doubt those jobs are being force deleted)
yeah, there was no force deletion involved in (or any deletion at all, actually... the completed pods just sat there, completed)
The set of admitted pods considered for pod count is everything in the pod manager filtered by filterOutTerminatingPods, which calls CouldHaveRunningContainers which returns true once a pod is seen by the workers (shortly after admission, and until we’ve confirmed via syncTerminatingPod that no containers are running). Whatever the problem is it’s likely in the interaction of that.
Code change:
return kl.PodResourcesAreReclaimed(pod, status) } <del> // filterOutTerminatedPods returns the pods that could still have running // containers </del> <ins> // filterOutTerminatedPods returns pods that are not in a terminal phase // or are known to be fully terminated. This method should only be used // when the set of pods being filtered is upstream of the pod worker, i.e. // the pods the pod manager is aware of. </ins> func (kl *Kubelet) filterOutTerminatedPods(pods []*v1.Pod) []*v1.Pod { <del> var filteredPods []*v1.Pod </del> <ins> filteredPods := make([]*v1.Pod, 0, len(pods)) </ins> for _, p := range pods { <del> if !kl.podWorkers.CouldHaveRunningContainers(p.UID) { </del> <ins> if kl.podWorkers.IsPodKnownTerminated(p.UID) { continue } if p.Status.Phase == v1.PodSucceeded || p.Status.Phase == v1.PodFailed { </ins> continue } filteredPods = append(filteredPods, p)

Related discussion:
text: Because the kubelet now considers terminating pods, it rejects those rapidly rescheduled pods. The follow up is more involved but requires the kubelet to allow pods to be potentially admitted that can't be run currently due to those terminating pods. See the linked issue.
Code change:
}, }, } <ins> if runtime.GOOS != "darwin" { // Skip this error case on darwin, see https://github.com/golang/go/issues/21512 errorCases["spec.timeZone: Invalid value: "AMERICA/new_york": unknown time zone AMERICA/new_york"] = batch.CronJob{ ObjectMeta: metav1.ObjectMeta{ Name:      "mycronjob", Namespace: metav1.NamespaceDefault, UID:       types.UID("1a2b3c"), }, Spec: batch.CronJobSpec{ Schedule:          "0 * * * *", TimeZone:          &timeZoneBadCasing, ConcurrencyPolicy: batch.AllowConcurrent, JobTemplate: batch.JobTemplateSpec{ Spec: batch.JobSpec{ Template: validPodTemplateSpec, }, }, }, } } </ins> for k, v := range errorCases { errs := ValidateCronJobCreate(&v, corevalidation.PodValidationOptions{})

Related discussion:
text: test withnot FAIL In a macos. do or golang upstrem disscusion. We use time.LoadLocation to validate timezone setting in object spec like cronjob.(It is ok) We also add a test try to validate is not a crroct value. This test is not true sometimes, for example, in default macos this is a true time_zone value.(maybe also true in other case insensitive os, but I have no device to try) At least we should get the test to pass on macos, I hope. <details</details<details</details<details</details<details</details<details</details<details</details
The label(s) cannot be applied, because the repository doesn't have them. <detailsIn response to : /sig test (I guess) Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing (I guess)
see
/remove-sig testing /sig apps
I was very surprised to find that this error only occurs on darwin, and does not occur on windows, although windows is also case-sensitive by default. When using , windows finally use (see ), but darwin and linux have more choices (see ) they will first try , using the method, which will call the system's open method to open , because darwin is case-insensitive by default, and this operation will succeed.
/triage accepted
Code change:
"os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Related discussion:
text: Since upgrading from Kubernetes to we no longer get memory metrics from the endpoint on . Example: In our previous version, we would get a response like the following that included utilisation metrics: Hit the endpoint of on a cluster running version . Kubernetes version (use ): Cloud provider or hardware configuration: Provisioned on AWS with OS (e.g: ): Flatcar Container Linux by Kinvolk 2765.2.6 (Oklo) Kernel (e.g. ): 5.10.43-flatcar
/sig node
Hi AlexLastwhich container runtimes are you using? containerd?
Hi AlexLast, I have test on v1.22 with containerd, I did not reproduce this issue. If memory limit is set in the container spec of the pod, /stats/summary will report availableBytes, if memory limit is not set, /stats/summary will not be reported. just like this: ! ! code is here:
/cc We have reports of missing memory metrics on 1.20.8 and 1.21.2 now... was there a cadvisor or similar change that happened?
also /cc as an FYI
If you are using cri-o, it looks like a similar issue to and (possibly an issue with cadvisor/the way cri-o integrates with cadvisor). This would also apply to the dockershim
I'm not able to repro on GKE v1.20.8-gke.700 on containerd
Can you please provide more info about your environment, container runtime etc and possible repro steps? Thanks!
/triage needs-information
Hi, First of all, I am sorry to cut in on you. I have the same issue with Kubelet, in my case with Kubeadm. Here you are the issue: Here you are my installation: Thanks in advance
^ quick summary of the above, appears to be single-node ubuntu (not sure what kernel?) containerd 1.4.6 k8s 1.21.2 kubeadm installer
It is one node. It is only for making easier where is the problem .
Code change:
"os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Related discussion:
text: Kubernetes version: 1.21.2 Cloud provider or hardware configuration: Bare metal (VMware) OS: Ubuntu 18.04.5 LTS Kernel: 4.15.0-147-generic Install tools: Kubeadm Network plugin and version (if this is a network-related bug): Weave 2.8.1 CRI: Containerd 1.4.6
Sure, we're setting the following flags for : Version: We deploy our clusters using , these missing metrics are for all pods in the cluster, including pods with memory requests and limits set.
Hitting the same issue on such kind of environments Then I tested and found no issue on environment
Besides memory metrics, we noticed a loss of network in pods. I commented at the block of container and the block of pod The loss of pods occurs at the same time as loss of pods
I created cluster as follows on 1.21.2 with kubetest containerd 1.4.6 on ubuntu. This is on older version of ubuntu 18.04 LTS on kernel I don't see memory metrics being lost here.
Can we have more details on exact kernel/OS image being used here? I can't seem to repro this on 1.21.2 with same containerd/ ubuntu version as mentioned in
Hi, may I know your output of network metrics in pods?
Hi, I have tested with two kernel versions: P.S: I have just updated my in order to make easier the test. You can use it (it is necessary to install and ). Thanks in advance
We tested with two kernel versions too.
I was able to repro finally -- it looks like missing key here is that you must be using the systemd cgroup driver. With this patch and starting cluster as follows:
Hi do you mean that if using systemd cgroup driver, this issue would happen. In my environment, both containerd and kubelet are using cgroupfs driver, but the issue still exists containerd config dump
Hi, Here you are my Containerd & Kubelet config: ` As far as I know, systemd cgroup driver should be configured in order to have a proper perfomance in Kubernetes. Is it a bug with Kubelet?
Code change:
"os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Related discussion:
text: Thanks in advance
Hi thanks for the response, I've found the following in the , so will have a go at setting that parameter
Thanks for all the info! I was only able to repro this on systemd cgroup driver as mentioned in It did not repro for me when using cgroupfs. This will need some additional investigation to figure out the root cause and why memory metrics are not being reported correctly and see if this is a kubelet or cAdvisor issue. /cc
Hi, Thanks . By the way, there is a new issue reporting the same behaviour:
After setting kubelet's log verbosity level to 10, I found logs like Is this related? And I checked that according the log, all the pod sandboxID are missing
Thanks I was able to confirm that switching to fixed the missing metrics. From a bit of research, it's my understanding that using the driver on a node with is not recommended - can you confirm?
By stepping into the code, I found that the container info returned by this func is nil, so that the will always be false. Thus will always be . That explains the last log has expected fields. But because of 1), this func cannot be called. There's really a label-related issue on cadvisor: (Thanks With the workaround, I can recover the missed metrics
/remove-triage needs-information /triage accepted
Thanks for digging deeper into this I'm still a bit confused on the relationship between this and systemd cgroup driver. Do the labels being missing have something to do with systemd cgroup driver being used? Also are you using a different socket path for containerd i.e. not the default as mentioned in ?
We didn't use the default but used and not specified
Interesting! In the test I made in the container-runtime-endpoint is set to default (i.e. ).
With debugging, I found that on kubernetes 1.21.2 when using different cgroup driver, the cgroupPath format of containers are different. : systemd + containerd:cgroupfs : systemd + containerd: systemd : cgroupfs + containerd: cgroupfs The would return a map using the path.Base of cgroupPath Later, we use containerID to check to add metrics. So only in the third case, the key of are . That's how the cgroup driver impacts the metrics.
Code change:
"os" "path/filepath" "runtime" <ins> "strings" </ins> "testing" "time"

Related discussion:
text: To sum up, and are two different cases that caused this issue. For the case, I think it's a bug we need fix: we should make sure the key of always be BTW do you know why the cgroupPath format are different when using different cgroup driver?
I believe it's expected that the cgroup naming will differ when using systemd cgroup driver vs cgroupfs. Kubelet internally has some logic for parsing and converting between the two different formats, e.g. and I left some comments on your proposed PR: Thanks for helping look into this issue!
/priority important-soon
Code change:
return lw.storage.WatchList(context.TODO(), lw.resourcePrefix, options.ResourceVersion, Everything) } <ins> // cacherWatch implements watch.Interface to return a single error type errWatcher struct { result chan watch.Event } func newErrWatcher(err error) *errWatcher { // Create an error event errEvent := watch.Event{Type: watch.Error} switch err := err.(type) { case runtime.Object: errEvent.Object = err case *errors.StatusError: errEvent.Object = &err.ErrStatus default: errEvent.Object = &unversioned.Status{ Status:  unversioned.StatusFailure, Message: err.Error(), Reason:  unversioned.StatusReasonInternalError, Code:    http.StatusInternalServerError, } } // Create a watcher with room for a single event, populate it, and close the channel watcher := &errWatcher{result: make(chan watch.Event, 1)} watcher.result <- errEvent close(watcher.result) return watcher } // Implements watch.Interface. func (c *errWatcher) ResultChan() <-chan watch.Event { return c.result } // Implements watch.Interface. func (c *errWatcher) Stop() { // no-op } </ins> // cacherWatch implements watch.Interface type cacheWatcher struct { sync.Mutex

Related discussion:
text: Basic auth should already have a big warning around it saying "not safe with browsers"
Maybe I'm missing something, but it seems deterministic to me. This is the uncached watch impl: once the resourceVersion parses, there is no path that returns a direct error, so all errors would have to be returned as an event via the ResultChan(). I don't see us reading from that until after we've already written a 200 and flushed:
Having read the docs, I was very surprised that "type": "ERROR" notices are sent as 200 (even over plain http), with the semantic code e.g. 410 only represented under inside. I see there was good motivation here to stick to 200, but it should be called out in the documentation. doesn't show at all how watch returns errors, and it even says "clients must handle the case by recognizing the status code 410 Gone" which really makes one expect an actual HTTP 410.
I agree this should be called out in the docs. Can you file a bug there and/or send a PR? I don't think this ~4 year old issue is the best place to track that. (Good detective work finding this, though!)