jiebi/CodeConvo huggingface dataset:
Code change:
err = testsuites.WaitForControllerVolumeResize(ctx, pvc, m.cs, csiResizeWaitPeriod) framework.ExpectNoError(err, "While waiting for PV resize to finish") <ins> ginkgo.By("Waiting for PVC resize to finish") pvc, err = testsuites.WaitForFSResize(ctx, pvc, m.cs) framework.ExpectNoError(err, "while waiting for PVC to finish") </ins> ginkgo.By("Waiting for all remaining expected CSI calls") err = wait.Poll(time.Second, csiResizeWaitPeriod, func() (done bool, err error) { <del> _, index, err := compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </del> <ins> var index int _, index, err = compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </ins> if err != nil { return true, err }

Related discussion:
text: Seen in pull-kubernetes-e2e-gce-cos-alpha-features Specific job failure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
from the error message I don't know if it refers to the sig-storage error or to the sig-auth one
/sig storage
/kind failing-test
/assign From : ... I am not sure if it's related:
is the culprit. cannot be used inside a . This only gets caught when the test actually runs, otherwise Ginkgo never sees the incorrect .
I don’t understand why sometimes it can get the pv by namebut sometimes it failed
but then is a sig-auth test , not storage , sorry /sig auth this seems that was recently in
/remove sig-storage
/assign /priority important-soon
/unassign
is still failed. seems not fixing the whole problem. /reopen feel free to close if the ci is green.
Reopened this issue. <detailsIn response to : is still failed. /reopen feel free to close if the ci is green. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
the failure now looks like it is just a sig storage issue:
cc
/assign looks like the new test is still an issue
I think this line is the problem: When the test creates the object, it may take time for the PV to be provisioned and to be bound to the PVC. So you should do another before this call so you can get the latest PVC object with the bound PV.
I opened to get PVC again before using .
Only do PersistentVolumeClaims.Get() is not enough, whether we need to wait for sometimes to wait it bound? because sometimes the testcase is successful and sometime is failed.
If the Pod was able to run, then the PVC has to be bound.
so I can't understand why sometimes the testcase is successful and sometimes is failed.
Code change:
err = testsuites.WaitForControllerVolumeResize(ctx, pvc, m.cs, csiResizeWaitPeriod) framework.ExpectNoError(err, "While waiting for PV resize to finish") <ins> ginkgo.By("Waiting for PVC resize to finish") pvc, err = testsuites.WaitForFSResize(ctx, pvc, m.cs) framework.ExpectNoError(err, "while waiting for PVC to finish") </ins> ginkgo.By("Waiting for all remaining expected CSI calls") err = wait.Poll(time.Second, csiResizeWaitPeriod, func() (done bool, err error) { <del> _, index, err := compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </del> <ins> var index int _, index, err = compareCSICalls(ctx, trackedCalls, test.expectedCalls, m.driver.GetCalls) </ins> if err != nil { return true, err }

Related discussion:
text: By default, the DelayBinding is false, it will wait for PVC bound during creation. This is weird.
It waits for the claim to be bound, but the test case still needs to refresh its pvc object.
Because of the refresh is async, so after the pvc was bound, we need to get the pvc again, isn't it?
Code change:
} // NodeAllocatableRoot returns the literal cgroup path for the node allocatable cgroup <del> func NodeAllocatableRoot(cgroupRoot, cgroupDriver string) string { root := ParseCgroupfsToCgroupName(cgroupRoot) nodeAllocatableRoot := NewCgroupName(root, defaultNodeAllocatableCgroupName) </del> <ins> func NodeAllocatableRoot(cgroupRoot string, cgroupsPerQOS bool, cgroupDriver string) string { nodeAllocatableRoot := ParseCgroupfsToCgroupName(cgroupRoot) if cgroupsPerQOS { nodeAllocatableRoot = NewCgroupName(nodeAllocatableRoot, defaultNodeAllocatableCgroupName) } </ins> if libcontainerCgroupManagerType(cgroupDriver) == libcontainerSystemd { return nodeAllocatableRoot.ToSystemd() }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: k8s 1.17.2 config kubelet with those parameters cgroupDriver: cgroupfs cgroupRoot: /docker cgroupsPerQOS: false kubelet log always prints those E0303 17:25:03. ] Failed to get system container stats for "/docker": failed to get cgroup stats for "/docker": failed to get container info for "/docker": unknown container "/docker" E0303 17:25:03. ] eviction manager: failed to construct signal: "allocatableMemory.available" error: system container "pods" not found in metrics What you expected to happen: kubelet does not print those error information How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Code change:
for _, extIP := range svcInfo.ExternalIPStrings() { staleServices.Insert(extIP) } <ins> for _, extIP := range svcInfo.LoadBalancerIPStrings() { staleServices.Insert(extIP) } </ins> } }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --In ipvs mode client traffic(UDP) to a pod behind LoadbalancerIP is blackholed after node having the pod is rebooted. The issue seems to be similar to the below iptable issue, since client continuously tries to connect to the loadbalancerIP (even before kubeproxy has applied all rules), and we see that stale conntracks entries remain even after rules are applied. After the stale conntrack entries are cleaned manually the traffic starts flowing : From the below code looks like ipvs currently clears stale conntrack entries for ExternalIPs but not LoadbalancerIP(?): ().String())-,for%20%2C%20extIP%20%3A%3D%20range%20svcInfo.ExternalIPStrings()%20%7B,-staleServices.Insert(extIP UDP traffic between client and pod(behind Loadbalancer IP) should have been established after node reboot. a POD acting as a UDP server and expose it through Loadbalancer service a UDP client to continuously send traffic to the LoadbalancerIP the node having the pod. after the new pod serving UDP is up, the UDP traffic from client would be blackholed(until stale conntrack entries are cleared). Kubernetes version (use ): 1.21.1 Cloud provider or hardware configuration: Manufacturer: Dell Inc. Product Name: PowerEdge R640 Version: Not Specified OS (e.g: ): SUSE Linux Enterprise Server 15 SP2 Kernel (e.g. ): Linux pool16-n108-wk16-n080 5.3.18-24.75.3..0.PTF.-default # 1 SMP Thu Sep 9 23:24:48 UTC 2021 () x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Calico v3.19.1-12- Others:
/sig network
/assign I'll try to reproduce this using but it may take some days.
Code change:
for _, extIP := range svcInfo.ExternalIPStrings() { staleServices.Insert(extIP) } <ins> for _, extIP := range svcInfo.LoadBalancerIPStrings() { staleServices.Insert(extIP) } </ins> } }

Related discussion:
text: yeah, ipvs kube-proxy doesn't have some of the latest "conntrack stale" fixes like , mainly because I'm not very familiar with IPVS and I can't know if all of them are needed or only some of them.
/triage accepted is it a matter of inserting externalIPs in conntrack?
No, when a node is rebootes and udp messages to a loadBalancerIP enters before kube-proxy is started, then invalid (UNREPLIED) conntrack entrys are created. They will black-hole traffic, and since traffic keeps coming they will never timeout. This is not easy to reproduce. I found it best to direct incoming traffic to a node (2), and have one endpoint pod on another node (4). Then reboot the node where traffic enters (and let the pod live) while traffic is sent continously. After reboot, check the conntrack on the rebooted node. In this example 3 out of 10 connections are ok. Those are the fortunate ones that didn't send a packet in the small time span from node start to kube-proxy start. (about 1 pkt/sec is sent on all 10 connections)
/area kube-proxy /area ipvs
Code change:
func (im *realImageGCManager) Start() { ctx := context.Background() go wait.Until(func() { <del> // Initial detection make detected time "unknown" in the past. var ts time.Time if im.initialized { ts = time.Now() } _, err := im.detectImages(ctx, ts) </del> <ins> _, err := im.detectImages(ctx, time.Now()) </ins> if err != nil { klog.InfoS("Failed to monitor images", "err", err) <del> } else { im.initialized = true </del> } }, 5*time.Minute, wait.NeverStop)

Related discussion:
text: when kubelet restart,all images which notused cleaned. but there is a flag in kubelet,,explainMinimum age for an unused image before it is garbage collected. Examples: '300ms', '10s' or '2h45m'. i set to ,it's means if a image not used,it will live 20m and then clean by kubeletif has disk pressure.if kubelet not restart, it work,but if restart ther is a bug, not work,image will be cleaneven if it's no enough time. work, image even if not used,it will live 20m and then clean by kubelet. 1make node trigger disk pressure. 2pull some image which not used by pod 3restart kubelet 4image cleaned by kubelet after around 10s i see the source code, and maybe there is a bug. in func,well detect image and update firstdetect time for every image. and in func,well check images livetime by firstDetected. if kubelet running normal,it work,because there is a func. in this line,,in first time,ts is zero. so image's firstDetected time is zero,and in freeSpace this check is always false,so image will be cleaned. maybe we can change to ,can fix this bug. <details</details<detailsNone </details<details</details<details</details<details</details<details</details
/sig node
/assign
/triage accepted /priority backlog initial assessment of priority, subject to change
Code change:
apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

Related discussion:
text: When installing the metrics-server to implement the API and the prometheus-adapter to implement the API, it is possible to make the OpenAPI handler fail to initialize with this error log in apiserver: This will result in kubectl discovery calls to /openapi/v2 to fail with a 503: The OpenAPI handler should not fail to initialize if both and are serving the same path for different API groups/versions. I can provide exact installation steps if needed, however, a high-level repro is: install metrics-server to serve install prometheus-adapter to serve ensure resourceRules is populated in the prometheus-adapter config so that resource metrics are also served by restarting apiserver will result in the OpenAPI failure and 503 responses to /openapi/v2 path No response <details</details<detailsAWS </details<details</details<details</details<details</details<details</details
Please let me know if I should reopen the issue in
/sig api-machinery
/sig instrumentation [metrics apis]
Would you mind taking a look when have time? Thank you^^ /triage accepted
I have a question. should be serving under . Why does an apiservice serving another gvk register paths under ? cc we've made the assumption that an APIService (such as metrics-server or prometheus-adapter) serves only gvks under its own registered group version. I'll do some more investigation but just wondering if prometheus doesn't follow that assumption?
the prometheus-adapter can also be configured to serve and replace the metrics-server by enabling these metrics, However, if the prometheus-adapter fronts and metrics-server fronts , it does seem like the correct behavior would be for prometheus-adapter to serve until an operator updates the APIService to reference prometheus-adapter rather than metrics-server. We're seeing this issue occur for upgrades between 1.27 and 1.28 so it does look this path duplication did not cause the OpenAPI handler to fail on 1.27.
I just upgraded from 1.27 to 1.28 and running into this problem. Some kubectl apply operations fail, and ArgoCD cannot load its custom resources. Any suggested workaround? Would removing the from the prometheus configuration resolve it? Or perhaps disable metrics-server?
Simply looking at the code diff between 1.27 and 1.28 I can't pinpoint the cause.
Code change:
apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

Related discussion:
text: Can I get your exact repro steps? suggests that the conflicting paths should be removed before the merge. We always call this function with
One other question is if this is reproducible by just applying a set of yaml/installations or does it require a cluster upgrade to trigger? Thanks!
hit same problem and we cannot install any other new helm plugins. maybe we need to move forward
Here are verbose repro instructions. An apiserver restart on 1.28 will induce the issue.
Hello. I'm still having some trouble reproducing although I'm not 100% sure if the configuration is correct. I'm trying to repro via kind. afaik installing via helm has prometheus adapter serving . seems to suggest that prometheus adapter is serving under . Applying the update to the configmap per the instructions didn't change the apiservice on my end. Not sure if this is my misconfiguration or intended. I've also tested via integration test that having two openapis serving the same path doesn't cause errors and one will get filtered out. A simple workaround is probably to delete the metrics-server if it is already superseded by the prometheus-adapter.
The is not set up in an resource for us, we only created an APIService with . When I go fetch the openAPI schema served up by the endpoint, it has schema information for in it if that is enabled using . If I remove then the schema fetched from the server doesn't have stuff in it, and the error is resolved. We have been keeping around because I think something uses it directly, or used to. Maybe ? That could have changed But in any case, this problem is quite severe when it happens so it would be nice to fix it as there may be others who do the same update and have to do some deep diving into the logs and all kinds of crazy diagnostics and guesswork to figure it out.
How does the get reached without an APIService? The resources would be unavailable for the apiserver to query and it'll be through something else? I see, adding resources for me doesn't populate the OpenAPI. I wonder if this could be cloud provider specific. I'm totally supportive of fixing, it's just quite difficult to reproduce on my part .
Hmm good question.
Code change:
apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

Related discussion:
text: In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
Code change:
apiServiceSuccess := &v1.APIService{ Spec: v1.APIServiceSpec{ <ins> Group:   "success", Version: "v1", </ins> Service: &v1.ServiceReference{Name: "dummy2"}, }, }

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
binary | sha1 hash | md5 hash ------ | --------- | -------- <del> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-releases/release/v1.3.0-alpha.2/kubernetes.tar.gz) | `305c8c2af7e99d463dbbe4208ecfe2b50585e796` | `aadb8d729d855e69212008f8fda628c0` </del> <ins> [kubernetes.tar.gz](https://storage.googleapis.com/kubernetes-release/release/v1.3.0-alpha.2/kubernetes.tar.gz) | `305c8c2af7e99d463dbbe4208ecfe2b50585e796` | `aadb8d729d855e69212008f8fda628c0` </ins> ## Changes since v1.3.0-alpha.1

Related discussion:
text: Perhaps a bug in
Thanks for catching this! I'm surprised it wasn't noticed earlier. It's an off by 1 (s) error. We pushed to gs://kubernetes-releases vs gs://kubernetes-release. The offending s has been removed. I'll clean up the release itself.
I thought this was right, but I think this is still an issue. I'm trying to release , and it's building wrong. I guess there were actually two problems; you caught one, but not the one I meant. This issue was about: should report itself as , but instead it reports itself as .
I'll look into this, since it's blocking alpha releases.
This seems contrary to , but doesn't have enough context for me to know why it was changed.
This was fixed by a long time ago.
Code change:
} func (c *dynamicResourceClient) Patch(name string, pt types.PatchType, data []byte, opts metav1.PatchOptions, subresources ...string) (*unstructured.Unstructured, error) { <ins> if len(name) == 0 { return nil, fmt.Errorf("name is required") } </ins> result := c.client.client. Patch(pt). AbsPath(append(c.makeURLSegments(name), subresources...)...).

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Providing an empty to the dynamic client's method results in a panic. What you expected to happen: An error should be returned on empty . How to reproduce it (as minimally and precisely as possible): Call with Anything else we need to know?: The expects a nonempty to be provided, but does not validate this. If an empty name is provided, this method will panic because it expects an , but the response will actually be an . Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Others: client-go 10.0.0
/sig api-machinery