jiebi/CodeConvo huggingface dataset:
Code change:
<ins> check process kube-proxy with pidfile /var/run/kube-proxy.pid group kube-proxy start program = "/etc/init.d/kube-proxy start" stop program = "/etc/init.d/kube-proxy stop" if does not exist then restart if failed port 10249 protocol HTTP request "/healthz" with timeout 10 seconds then restart </ins>

Related discussion:
text: We already "healthcheck" kubelet and docker periodically, but miss one for kube-proxy.
It would make sense to do (only bind to localhost) at the same time we add the healthchecking configuration.
Code change:
} } for _, c := range pod.Spec.Containers { <del> if c.SecurityContext == nil { </del> <ins> if c.SecurityContext == nil || c.SecurityContext.Privileged == nil { </ins> continue } if *c.SecurityContext.Privileged {

Related discussion:
text: <!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): panic apiserver invalid memory address Is this a BUG REPORT or FEATURE REQUEST? (choose one): Bug Report <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: aws OS (e.g. from /etc/os-release): Ubuntu 16.04 Kernel (e.g. ): Linux ip-10-50-193-54 4.4.0-59-generic -Ubuntu SMP Fri Jan 6 17:47:47 UTC 2017 x8664 x8664 x86_64 GNU/Linux Install tools: Others: What happened: Tried to exec: . Got back: apiserver logs: It only seems to happen on this one pod - other pods are fine. What you expected to happen: should succeed or at least give a sensible error message. How to reproduce it (as minimally and precisely as possible): Anything else do we need to know:
There are no errors on the node with the pod. I also tried kubectl v1.4.8 and same problem.
It looks like your pod that's having the issue has a container with a security context, but privileged appears to be nil. Can you confirm that is the case?
Yep that's it. It seems when securityContext is specified without setting , it isn't set correctly to its default of false. e.g. the pod yaml has: Explicitly adding allows exec to succeed.
Ah, this was caused by the kubelet using versioned APIs - for his awareness. Basically any code that depends on nil pointers being set is vulnerable.
Code change:
{CPU: 2000, Memory: 3000}, }, }, <ins> Host: "machine2", </ins> } tests := []struct { pod          api.Pod

Related discussion:
text: When creating replication controller (e.g. frontend php-redis from guestbook example), scheduler gets stuck and never recovers. This is a race, I have to run several times to see the error (A script to delete controller and pods, then create new controller, leave enough time in between for k8s to react). I've created PR for an attempted fix.
I saw this too while following the guest book and creating the php/frontend step. In my case, I had 2 minions while the replication controller configuration had configured 3 replicas, so 2 pods got scheduled -each on another minion, the third pod had empty Host (saw that when doing 'list pods'), and the 3 of them were stuck in "waiting" status. I deleted the controller and the pods, fixed the configuration in json file to be 2 replicas and recreated the controller and everything worked. Does your fix take care of this use case? (too many replicas, too few minions/one of the pods is not scheduled)
This is different from your use case. If you have more replicas than minions, then the scheduler will get stuck for sure (with the error msg probably being 'failed to find a fit for pod'). AFAICT, it's not support yet.
If you remove the HostPorts, it should work for you. thanks for report.
, thanks for your replies. I understand why it isn't working, but I'd expect one of the 2 options to happen instead of the current situation: the creation of replication controller if there are less minions than replicas, AND there is a hostPort property. In other words - validation of the content during creation. least schedule all replicas that can get a minion (in my case 2 out of 3) and leave only one at "waiting". Is there an issue open about the use case I describe?
should be true today. is less possible than it seems like-- minions can come and go, and become over or under loaded in the time between creation and when the pod shows up at the scheduler. So even if we check up front, we still have to check again later, and sometimes the first check will pass but not the second.
Code change:
Freshness: apidiscoveryv2beta1.DiscoveryFreshnessCurrent, }) } <ins> sort.Sort(byVersion(aggregatedVersions)) </ins> aggregatedDiscovery := []apidiscoveryv2beta1.APIGroupDiscovery{{ ObjectMeta: metav1.ObjectMeta{ Name: resources["v1"].Group, }, Versions: aggregatedVersions, }} <del> require.Equal(t, aggregatedDiscovery, doc.Items) </del> <ins> require.Equal(t, doc.Items, aggregatedDiscovery) </ins> } <ins> type byVersion []apidiscoveryv2beta1.APIVersionDiscovery var versionMap = map[string]int{ "v1":       1, "v1beta1":  2, "v1alpha1": 3, } func (a byVersion) Len() int           { return len(a) } func (a byVersion) Swap(i, j int)      { a[i], a[j] = a[j], a[i] } func (a byVersion) Less(i, j int) bool { return versionMap[a[i].Version] < versionMap[a[j].Version] } </ins> func TestLegacyFallback(t *testing.T) { aggregatedResourceManager := discoveryendpoint.NewResourceManager() rootAPIsHandler := discovery.NewRootAPIsHandler(discovery.DefaultAddresses{DefaultAddress: "192.168.1.1"}, scheme.Codecs)

Related discussion:
text: ! /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
It should be fixed by
/sig api-machinery
/triage accepted
Code change:
if !isPodManagedContainer(&info) { continue } <del> stats[path.Base(key)] = info </del> <ins> stats[extractIDFromCgroupPath(key)] = info </ins> } return stats } <ins> func extractIDFromCgroupPath(cgroupPath string) string { // case0 == cgroupfs: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32" id := path.Base(cgroupPath) // case1 == systemd: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope" // trim anything before the final '-' and suffix .scope systemdSuffix := ".scope" if strings.HasSuffix(id, systemdSuffix) { id = strings.TrimSuffix(id, systemdSuffix) components := strings.Split(id, "-") if len(components) > 1 { id = components[len(components)-1] } } return id } </ins>

Related discussion:
text: Since upgrading from Kubernetes to we no longer get memory metrics from the endpoint on . Example: In our previous version, we would get a response like the following that included utilisation metrics: Hit the endpoint of on a cluster running version . Kubernetes version (use ): Cloud provider or hardware configuration: Provisioned on AWS with OS (e.g: ): Flatcar Container Linux by Kinvolk 2765.2.6 (Oklo) Kernel (e.g. ): 5.10.43-flatcar
/sig node
Hi AlexLastwhich container runtimes are you using? containerd?
Hi AlexLast, I have test on v1.22 with containerd, I did not reproduce this issue. If memory limit is set in the container spec of the pod, /stats/summary will report availableBytes, if memory limit is not set, /stats/summary will not be reported. just like this: ! ! code is here:
/cc We have reports of missing memory metrics on 1.20.8 and 1.21.2 now... was there a cadvisor or similar change that happened?
also /cc as an FYI
If you are using cri-o, it looks like a similar issue to and (possibly an issue with cadvisor/the way cri-o integrates with cadvisor). This would also apply to the dockershim
I'm not able to repro on GKE v1.20.8-gke.700 on containerd
Can you please provide more info about your environment, container runtime etc and possible repro steps? Thanks!
/triage needs-information
Hi, First of all, I am sorry to cut in on you. I have the same issue with Kubelet, in my case with Kubeadm. Here you are the issue: Here you are my installation: Thanks in advance
^ quick summary of the above, appears to be single-node ubuntu (not sure what kernel?) containerd 1.4.6 k8s 1.21.2 kubeadm installer
It is one node. It is only for making easier where is the problem .
Code change:
if !isPodManagedContainer(&info) { continue } <del> stats[path.Base(key)] = info </del> <ins> stats[extractIDFromCgroupPath(key)] = info </ins> } return stats } <ins> func extractIDFromCgroupPath(cgroupPath string) string { // case0 == cgroupfs: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32" id := path.Base(cgroupPath) // case1 == systemd: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope" // trim anything before the final '-' and suffix .scope systemdSuffix := ".scope" if strings.HasSuffix(id, systemdSuffix) { id = strings.TrimSuffix(id, systemdSuffix) components := strings.Split(id, "-") if len(components) > 1 { id = components[len(components)-1] } } return id } </ins>

Related discussion:
text: Kubernetes version: 1.21.2 Cloud provider or hardware configuration: Bare metal (VMware) OS: Ubuntu 18.04.5 LTS Kernel: 4.15.0-147-generic Install tools: Kubeadm Network plugin and version (if this is a network-related bug): Weave 2.8.1 CRI: Containerd 1.4.6
Sure, we're setting the following flags for : Version: We deploy our clusters using , these missing metrics are for all pods in the cluster, including pods with memory requests and limits set.
Hitting the same issue on such kind of environments Then I tested and found no issue on environment
Besides memory metrics, we noticed a loss of network in pods. I commented at the block of container and the block of pod The loss of pods occurs at the same time as loss of pods
I created cluster as follows on 1.21.2 with kubetest containerd 1.4.6 on ubuntu. This is on older version of ubuntu 18.04 LTS on kernel I don't see memory metrics being lost here.
Can we have more details on exact kernel/OS image being used here? I can't seem to repro this on 1.21.2 with same containerd/ ubuntu version as mentioned in
Hi, may I know your output of network metrics in pods?
Hi, I have tested with two kernel versions: P.S: I have just updated my in order to make easier the test. You can use it (it is necessary to install and ). Thanks in advance
We tested with two kernel versions too.
I was able to repro finally -- it looks like missing key here is that you must be using the systemd cgroup driver. With this patch and starting cluster as follows:
Hi do you mean that if using systemd cgroup driver, this issue would happen. In my environment, both containerd and kubelet are using cgroupfs driver, but the issue still exists containerd config dump
Hi, Here you are my Containerd & Kubelet config: ` As far as I know, systemd cgroup driver should be configured in order to have a proper perfomance in Kubernetes. Is it a bug with Kubelet?
Code change:
if !isPodManagedContainer(&info) { continue } <del> stats[path.Base(key)] = info </del> <ins> stats[extractIDFromCgroupPath(key)] = info </ins> } return stats } <ins> func extractIDFromCgroupPath(cgroupPath string) string { // case0 == cgroupfs: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32" id := path.Base(cgroupPath) // case1 == systemd: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope" // trim anything before the final '-' and suffix .scope systemdSuffix := ".scope" if strings.HasSuffix(id, systemdSuffix) { id = strings.TrimSuffix(id, systemdSuffix) components := strings.Split(id, "-") if len(components) > 1 { id = components[len(components)-1] } } return id } </ins>

Related discussion:
text: Thanks in advance
Hi thanks for the response, I've found the following in the , so will have a go at setting that parameter
Thanks for all the info! I was only able to repro this on systemd cgroup driver as mentioned in It did not repro for me when using cgroupfs. This will need some additional investigation to figure out the root cause and why memory metrics are not being reported correctly and see if this is a kubelet or cAdvisor issue. /cc
Hi, Thanks . By the way, there is a new issue reporting the same behaviour:
After setting kubelet's log verbosity level to 10, I found logs like Is this related? And I checked that according the log, all the pod sandboxID are missing
Thanks I was able to confirm that switching to fixed the missing metrics. From a bit of research, it's my understanding that using the driver on a node with is not recommended - can you confirm?
By stepping into the code, I found that the container info returned by this func is nil, so that the will always be false. Thus will always be . That explains the last log has expected fields. But because of 1), this func cannot be called. There's really a label-related issue on cadvisor: (Thanks With the workaround, I can recover the missed metrics
/remove-triage needs-information /triage accepted
Thanks for digging deeper into this I'm still a bit confused on the relationship between this and systemd cgroup driver. Do the labels being missing have something to do with systemd cgroup driver being used? Also are you using a different socket path for containerd i.e. not the default as mentioned in ?
We didn't use the default but used and not specified
Interesting! In the test I made in the container-runtime-endpoint is set to default (i.e. ).
With debugging, I found that on kubernetes 1.21.2 when using different cgroup driver, the cgroupPath format of containers are different. : systemd + containerd:cgroupfs : systemd + containerd: systemd : cgroupfs + containerd: cgroupfs The would return a map using the path.Base of cgroupPath Later, we use containerID to check to add metrics. So only in the third case, the key of are . That's how the cgroup driver impacts the metrics.
Code change:
if !isPodManagedContainer(&info) { continue } <del> stats[path.Base(key)] = info </del> <ins> stats[extractIDFromCgroupPath(key)] = info </ins> } return stats } <ins> func extractIDFromCgroupPath(cgroupPath string) string { // case0 == cgroupfs: "/kubepods/burstable/pod2fc932ce-fdcc-454b-97bd-aadfdeb4c340/9be25294016e2dc0340dd605ce1f57b492039b267a6a618a7ad2a7a58a740f32" id := path.Base(cgroupPath) // case1 == systemd: "/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc932ce_fdcc_454b_97bd_aadfdeb4c340.slice/cri-containerd-aaefb9d8feed2d453b543f6d928cede7a4dbefa6a0ae7c9b990dd234c56e93b9.scope" // trim anything before the final '-' and suffix .scope systemdSuffix := ".scope" if strings.HasSuffix(id, systemdSuffix) { id = strings.TrimSuffix(id, systemdSuffix) components := strings.Split(id, "-") if len(components) > 1 { id = components[len(components)-1] } } return id } </ins>

Related discussion:
text: To sum up, and are two different cases that caused this issue. For the case, I think it's a bug we need fix: we should make sure the key of always be BTW do you know why the cgroupPath format are different when using different cgroup driver?
I believe it's expected that the cgroup naming will differ when using systemd cgroup driver vs cgroupfs. Kubelet internally has some logic for parsing and converting between the two different formats, e.g. and I left some comments on your proposed PR: Thanks for helping look into this issue!
/priority important-soon
Code change:
"[Alpha]", }, { <del> "a test case with multiple invalid tags [Flaky][Disruptive] [Feature:Awesome] [Alpha]", "[Flaky],[Disruptive],[Feature:Awesome],[Alpha]", </del> <ins> "a test case with multiple invalid tags [Flaky] [Feature:Awesome] [Alpha]", "[Flaky],[Feature:Awesome],[Alpha]", </ins> }, { <del> "[sig-awesome] [Disruptive] a test case with valid and invalid tags [Alpha] [Serial] [Flaky]", "[Disruptive],[Alpha],[Flaky]", </del> <ins> "[sig-awesome] [Alpha] [Disruptive] a test case with valid and invalid tags [Serial] [Flaky]", "[Alpha],[Flaky]", </ins> }, } for i, tc := range testCases {

Related discussion:
text: What would you like to be : Our tooling that makes the list of conformance tests needs to allow conformance tests to be marked as disruptive. Why is this needed: To test default, conformance behavior which may be disruptive (e.g. taint eviction) ref /sig testing /area conformance
Code change:
"[Alpha]", }, { <del> "a test case with multiple invalid tags [Flaky][Disruptive] [Feature:Awesome] [Alpha]", "[Flaky],[Disruptive],[Feature:Awesome],[Alpha]", </del> <ins> "a test case with multiple invalid tags [Flaky] [Feature:Awesome] [Alpha]", "[Flaky],[Feature:Awesome],[Alpha]", </ins> }, { <del> "[sig-awesome] [Disruptive] a test case with valid and invalid tags [Alpha] [Serial] [Flaky]", "[Disruptive],[Alpha],[Flaky]", </del> <ins> "[sig-awesome] [Alpha] [Disruptive] a test case with valid and invalid tags [Serial] [Flaky]", "[Alpha],[Flaky]", </ins> }, } for i, tc := range testCases {

Related discussion:
text: What happened: Ran a conformance test run (using Sonobuoy) on One of the taint based tests a NoExecute taint to a node which evicted all workloads including the Sonobuoy workload itself. If this had been a prod workload, numerous prod workloads could have been impacted. What you expected to happen: Test run as normal with reasonable results. Able to run conformance tests with minimal impact on a running cluster. How to reproduce it (as minimally and precisely as possible): on a v1.16.0 cluster. Anything else we need to know?: Environment: Kubernetes version (use ): /area conformance
/sig testing /sig architecture /sig scheduling
Summarizing the many conversations on the PR and in Slack regarding this: We wish we had identified the lack of a tag sooner and could have taken better action. Since the release is 2 days away and our own tooling currently prohibits us from marking tests as both and , we've decided to let the release proceed with this "bug". Risk: Someone running the conformance tests may unexpectedly have their workloads evicted Mitigation: Sonobuoy v0.16.0 will provide the capabilities to the user to run a "conformance-lite" (name TBD) run if they'd like to run the conformance tests against a cluster with prod workloads. This way end-users can still be reasonably sure of their cluster's continued well-being. This mode will manually skip the tests in question until they can be more appropriately marked as Sonobuoy will also provide, consistent with past behavior, the means to run the full conformance suite including these disruptive tests. Eventual solution: Sonobuoy will, most likely, need to continue offering a "non-disruptive-conformance" mode to aid in end user testing but the regexp will change and should be able to just skip tests K8s tooling needs to be tweaked to allow for tests to be marked as The tag should be to these tests.
Code change:
// Try to unmap podUID symlink under global map path dir // plugins/kubernetes.io/{PluginName}/volumeDevices/{volumePluginDependentPath}/{podUID} globalUnmapPath := volumeToUnmount.DeviceMountPath <del> if err != nil { // On failure, return error. Caller will log and retry. return volumeToUnmount.GenerateError("UnmapVolume.GetGlobalUnmapPath failed", err) } </del> unmapDeviceErr = og.blkUtil.UnmapDevice(globalUnmapPath, string(volumeToUnmount.PodUID)) if unmapDeviceErr != nil { // On failure, return error. Caller will log and retry.

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: A Pod is run with local volume mounted in block mode. When that pod is deleted, the tear down process fails because device unmap fails. The device path appears to be empty, causing the unmap to fail. What you expected to happen: The teardown should have succeeded How to reproduce it (as minimally and precisely as possible): Create a pod with a PV for local volume in block mode. Delete the pod. Look at kubelet log files on the minion node to observe the error messages. Anything else we need to know?: Log messages: Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: E0218 09:21:43. 1510 ] Operation for """ ("-148c-11e8-b315-")" failed. No retries permitted until 2018-02-18 09:21:44. +0000 UTC m=+. (durationBeforeRetry 500ms). Error: "UnmapVolume.UnmapDevice on global map path failed for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") : Failed to unmap device from map path. mapPath is empty" Feb 18 09:21:43 kubernetes-minion-group-crj1 kubelet[1510]: I0218 09:21:43. 1510 ] operationExecutor.UnmountVolume started for volume "blkdata" (UniqueName: "") pod "-148c-11e8-b315-" (UID: "-148c-11e8-b315-") Environment: Kubernetes version (use ): 1.10+ Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): "Container-Optimized OS from Google" Kernel (e.g. ): Linux 4.4.86+ Install tools: Others:
/sig storage
This error is returned in this . It same that we set the global mount path to only if the volume plugin is attachable(see ).
Code change:
// Try to unmap podUID symlink under global map path dir // plugins/kubernetes.io/{PluginName}/volumeDevices/{volumePluginDependentPath}/{podUID} globalUnmapPath := volumeToUnmount.DeviceMountPath <del> if err != nil { // On failure, return error. Caller will log and retry. return volumeToUnmount.GenerateError("UnmapVolume.GetGlobalUnmapPath failed", err) } </del> unmapDeviceErr = og.blkUtil.UnmapDevice(globalUnmapPath, string(volumeToUnmount.PodUID)) if unmapDeviceErr != nil { // On failure, return error. Caller will log and retry.

Related discussion:
text: However, local volume plugin is not attachable, so will be empty and the operation will fail. I think we should extract the operation from the block, and execute this operation just before we try to map the device to global map path, then this global map path can be cached in for both attachable and non attachable volume plugins. Any comments? If not, I can help to fix this.
Thank you for the investigation. We've discussed this problem by offline yesterday. I proposed same fix which you mentioned above, then tested the fix works correctly with local volume plugin. I'll push a PR soon.
The device path appears to be empty, causing the unmap to fail. Please note, which was stored in is empty. device path isn't related to this problem.
Sorry about that, this is a typo, what I mean is global mount path.
No, no. My above comment is just note for the section. Your investigation is correct.
Code change:
PromoterE2eRegistry     string `yaml:"promoterE2eRegistry"` BuildImageRegistry      string `yaml:"buildImageRegistry"` InvalidRegistry         string `yaml:"invalidRegistry"` <del> GcHttpdRegistry         string `yaml:"gcHttpdRegistry"` </del> GcEtcdRegistry          string `yaml:"gcEtcdRegistry"` GcRegistry              string `yaml:"gcRegistry"` SigStorageRegistry      string `yaml:"sigStorageRegistry"`

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --/sig windows /cc After was merged I think.
I agree that looks like the issue. It started failing between the two commits: on this run:
/triage accepted
/assign