jiebi/CodeConvo huggingface dataset:
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

Related discussion:
text: In our staging cluster I tried bringing back the in the custom metrics server configuration and restarting all the masters, and the problem did not come back, everything seems OK now. So it maybe isn't that easy to reproduce
After updating the configmap, restarting the prometheus-adapter pod to pickup the changes and restarting the apiserver are both required to induce the issue. I have been able to deterministically reproduce this following those steps, please follow up here if you are still having issues.
That's good to hear. I'll need some more help with reproducing it. I've been able to verify that restarting the prometheus adapter pod picks up the configmap changes but restarting the apiserver hasn't caused any crashes for me. I've tested this both on the latest master branch and 1.28.
While I'm yet to fully understand these changes, the 2 PRs below (both of which went into 1.28) seem likely candidates for a regression here: - introduced lazy loading in openAPI v2 handler to load/compute the aggregator spec when requested by client - switched the spec merging function from to any clue if one of the above changes could be causing mischief?
Also from example request at the top, the impact seems to be to the v2 API: Nathan - when you get a chance, could you also confirm if v1 API doesn't run into this?
OpenAPI only supports v2 and v3.
hi, any clue about this issue?
It's possible but I haven't been able to reproduce it. Can I ask if this is happening on cloud providers outside AWS?
Here are the apiservices and relevant pod objects with a working repro from these steps If it's useful, I can provide yamls for these objects in addition to the settings applied in the prometheus-adapter configmap. Can you confirm that the pods backing your prometheus-adapter and metrics-server are running?
I meet this problem
We the problem in GCP as well.
ACK in Alibaba Cloud also meets this problem
We were able to fix the v2 availability by and re-installing the kube-prometheus helm chart. Unfortunately, after restarting the k8s api server, v2 availability is broken again.
yes, I'm running on kind. I'll do some more digging, assuming it's related to those PRs linked by but still haven't pinpointed anything.
Code change:
SwaggerProps: spec.SwaggerProps{ Paths: &spec.Paths{ Paths: map[string]spec.PathItem{ <del> "/apis/apiservicegroup/v2": {}, </del> <ins> "/apis/apiservicegroup/v1/path2": {}, </ins> }, }, },

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
import ( "context" <ins> "errors" </ins> "fmt" "os" "path/filepath"

Related discussion:
text: treats any error returned by the condition as terminal, but silently ignores any errors that aren't returned along with . We should change WaitForPodCondition to behave like . /sig testing /area e2e-test-framework
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
Was there any reason why you instead of doing this right away? I fully agree with this issue, we should abort early. I just found the TODO because I have a failing test that ran longer than expected after a final pod failure and that made me check out why it didn't abort quicker :grin:
Code change:
} return result } <ins> // Verifies the specified metrics are in `kubeletMetrics` func verifyVolumeStatMetric(metricKeyName string, namespace string, pvcName string, kubeletMetrics metrics.KubeletMetrics) { found := false errCount := 0 if samples, ok := kubeletMetrics[metricKeyName]; ok { for _, sample := range samples { samplePVC, ok := sample.Metric["persistentvolumeclaim"] if !ok { framework.Logf("Error getting pvc for metric %s, sample %s", metricKeyName, sample.String()) errCount++ } sampleNS, ok := sample.Metric["namespace"] if !ok { framework.Logf("Error getting namespace for metric %s, sample %s", metricKeyName, sample.String()) errCount++ } if string(samplePVC) == pvcName && string(sampleNS) == namespace { found = true break } } } Expect(errCount).To(Equal(0), "Found invalid samples") Expect(found).To(BeTrue(), "PVC %s, Namespace %s not found for %s", pvcName, namespace, metricKeyName) } </ins>

Related discussion:
text: In 1.8, we a new feature to expose PVC information through summary API so user can check the volume belongs to which PVC. We should add e2e test in under test/e2enode.
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
[MILESTONENOTIFIER] Milestone Labels Complete Issue label settings: sig/storage: Issue will be escalated to these SIGs if needed. priority/important-soon: Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. kind/cleanup: Adding tests, refactoring, fixing old bugs. <detailsAdditional instructions available <a href=""here</aThe commands available for adding these labels are documented <a href=""here</a</details
/assign
GitHub didn't allow me to assign the following users: vkamra. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Can we please move this out of 1.8 milestone?
- I'm looking into adding this but yes we should move this out of 1.8. I don't have permissions to do that though.
ack Can we please move this out of 1.8?
Sorry am not sure why was this removed from 1.8 milestone. Are we not aiming to add the tests in 1.8? Or Are we disabling the feature or are we fine shipping it without any e2e tests in 1.8?
The test is in . I think we can keep this feature in 1.8
Code change:
framework.ExpectEqual(len(podTemplateList.Items), 0, "PodTemplate list returned items, failed to delete PodTemplate") }) <del> ginkgo.It("should delete a collection of pod templates", func() { </del> <ins> /* Release : v1.19 Testname: PodTemplate, delete a collection Description: A set of Pod Templates is created with a label selector which MUST be found when listed. The set of Pod Templates is deleted and MUST NOT show up when listed by its label selector. */ framework.ConformanceIt("should delete a collection of pod templates", func() { </ins> podTemplateNames := []string{"test-podtemplate-1", "test-podtemplate-2", "test-podtemplate-3"} ginkgo.By("Create set of pod templates")

Related discussion:
text: [x] Original approval issue: [x] two weeks soak end date: 2 July 2020 This issue is created to allow edit by According to this APIsnoop query, there are still an endpoint which is untested. /wip /hold a set of 3 Pod Templates with a static label within a Namespace. that all 3 Pod Templates with the label in the Namespace are created. the set of Namespaced Pod Templates with a label via DeleteCollection. that all Pod Templates with the label have been deleted. Discover useragents: List endpoints hit by the test: Display endpoint coverage change: If a test with these calls gets merged, test coverage will go up by 1 points This test is also created with the goal of conformance promotion.
/sig testing /sig architecture /area conformance
/area conformance
During Conformance Meeting: Is there a way to write this test so it is resource agnostic? If we are just changing the resource... can be parameterize the resource-type
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
GitHub didn't allow me to assign the following users: heyste. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/reopen Merge 17 July 2020
Reopened this issue. <detailsIn response to : /reopen Merge 17 July 2020 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
var wasGracePeriodShortened bool switch { case status.IsTerminated(): <ins> // A terminated pod may still be waiting for cleanup - if we receive a runtime pod kill request // due to housekeeping seeing an older cached version of the runtime pod simply ignore it until // after the pod worker completes. if isRuntimePod { klog.V(3).InfoS("Pod is waiting for termination, ignoring runtime-only kill until after pod worker is fully terminated", "pod", klog.KObj(pod), "podUID", pod.UID) return } </ins> workType = TerminatedPodWork if options.KillPodOptions != nil {

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --When testing against OpenShift CI, I discovered a panic in the kubelet: Further investigation revealed that it is possible for to panic under certain circumstances: We hit this case when (podStatus remains nil) and (nil will be passed along to ). I'm not sure why exactly that would happen but it seems possible. Don't panic! Unclear, I've been trying to trace the code/find evidence of this happening in k8s CI but haven't found a match yet. ( would be nice :smile:) /sig node /cc /priority important-soon /milestone v1.22 Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
/triage accepted
/assign
The bug is that running pods should terminate and then stop, not move to terminated. So something is getting into terminated but should never be in any other work type than terminating. We'll need more debug to determine why but I suspect that an orphaned pod is still in podSyncStatus at the time it UpdatePod is called, which means it's not seen as being in Terminating. isRuntimePod forces podSyncStatus to be a certain state, but perhaps it's should resett a terminated bool. Asked elana to add a v4 min level in the test jobs that reproduced this so we can see.
has bumped, here's a v=4 log with a panic: https://gcsweb-
Elana got the version with more logs and it is indeed the case that the pod is terminating (in this case because a CSI volume hasn't been torn down).
Code change:
var wasGracePeriodShortened bool switch { case status.IsTerminated(): <ins> // A terminated pod may still be waiting for cleanup - if we receive a runtime pod kill request // due to housekeeping seeing an older cached version of the runtime pod simply ignore it until // after the pod worker completes. if isRuntimePod { klog.V(3).InfoS("Pod is waiting for termination, ignoring runtime-only kill until after pod worker is fully terminated", "pod", klog.KObj(pod), "podUID", pod.UID) return } </ins> workType = TerminatedPodWork if options.KillPodOptions != nil {

Related discussion:
text: So the bug is: worker for the pod is still running (terminating has completed, but is executing syncTerminatedPod) runs, calls , gets a stale result for a pod (thinks the pod is still running, but that's just a stale cache result) asks pod worker to terminate the pod with a runningPod worker sees that the pod is actually not terminating, but terminated, and so RunningPod is inconsistently handled (UpdatePod is not defensive enough) worker passes the kill pod request through to syncTerminatedPod which has a null pod status, causing the panic In earlier versions of we explicitly refreshed the runtime cache in , but when and I went through review we removed it and I forgot this race was possible. The race becomes more possible here because the particular workload Elana is testing involves CSI drivers that have a bad cleanup loop, so syncTerminatedPod is reaching timeout on volume unmount (3m) before finally succeeding. This means the race window is huge (invoke UpdatePod(runningPod) on an terminated but not yet cleaned up pod). I think the real bug is that UpdatePod is not defensive enough - the difference between this race condition (seeing an older runtime container) and a human somehow starting the container up again before it is terminated is pretty similar. The important difference is that the UpdatePod{... running pod } is intended to clean up orphaned pods - those the pod worker is not tracking. But we only go into terminated when we are certain the workload is stopped, so if we simply ignore until the pod goes away it might be a while before we kill that container. We absolutely need a background worker tracking that, which is best to be a pod worker, but it's a bit more work. So easy fix (remove the possibility of the very common race) is to simply have syncTerminatingPod refresh the pod cache for that one pod. A slightly less easy fix is to have UpdatePod simply ignore the runningPod request until the next housekeeping loop (which in some cases with bad csi drivers could be a few minutes or so and in the worst case might block). The most correct fix is to ensure syncTerminated pod has an opportunity to maintain its invariants (no running containers) or have that be maintained alongside. Pondering variations here of fixes.
Code change:
var wasGracePeriodShortened bool switch { case status.IsTerminated(): <ins> // A terminated pod may still be waiting for cleanup - if we receive a runtime pod kill request // due to housekeeping seeing an older cached version of the runtime pod simply ignore it until // after the pod worker completes. if isRuntimePod { klog.V(3).InfoS("Pod is waiting for termination, ignoring runtime-only kill until after pod worker is fully terminated", "pod", klog.KObj(pod), "podUID", pod.UID) return } </ins> workType = TerminatedPodWork if options.KillPodOptions != nil {

Related discussion:
text: The easy fixes will defiintely be "safe", but "slight regression in latency in a few rare cases". Serializing the pod worker caused this, but it's in theory rare.
My understanding is that we make pod worker the source of truth for pod state. In this kind of race condition, as long as the request is sent to the pod worker, and the pod UID matches, it should be able to reconcile the state. Even if we do this, it is not atomic, the race condition still exists in theory. We probably should make the pod worker be able to reconcile the state. I don't quite understand what you mean by "until the next housekeeping loop"? Since the pod is being terminated in , what problem would it cause if we just ignore the runningPod request? If the termination is still in progress, we need to wait for it to finish anyway.
Here's the most concerning scenario: is terminated and then deleted from apiserver kills the container is taking a very long time after 2, housekeeping fires and sees the stale cache, triggers a kill (proposed change) ignore the kill request (since terminated pod is running) restarts container in the pod that references a volume mount can't make progress because volume detach can't make progress because the restarted container is holding a device lock / has a file open / w/e pod worker for that pod retries forever is the minimal fix that ignores (5) but I'm thinking through how to handle 8. I don't think we should do the easy fix (refreshing the cache isn't enough because you are vulnerable to 6-8). I'm potentially considering having the pod worker invoke syncTerminatingPod in the event syncTerminatedPod exits with an error, or simply have syncTerminatedPod do it if it would return an error to address 8.
But this basically means that after , there could still be running containers. In that case, this wouldn't be the only problem, the whole assumption the refactor builds on becomes invalid? BTW, restarting container may happen to docker, but containerd doesn't support it (at least it can't be done easily), not sure about cri-o.
cri-o doesn't support restarting a container, either.
The assumption in the kubelet is there are no more containers started by the kubelet, which is indeed safe.
Code change:
var wasGracePeriodShortened bool switch { case status.IsTerminated(): <ins> // A terminated pod may still be waiting for cleanup - if we receive a runtime pod kill request // due to housekeeping seeing an older cached version of the runtime pod simply ignore it until // after the pod worker completes. if isRuntimePod { klog.V(3).InfoS("Pod is waiting for termination, ignoring runtime-only kill until after pod worker is fully terminated", "pod", klog.KObj(pod), "podUID", pod.UID) return } </ins> workType = TerminatedPodWork if options.KillPodOptions != nil {

Related discussion:
text: That doesn’t prevent bugs in CRI implementations, user error, cosmic rays, basically computers. It’s unlikely and not something we really designed for before, but the prerefactor code would survive it. Appears to have gotten past the issue as expected. I’ll look at what it would take to mitigate the possibility of zombie containers such that the pod worker converges even if volume mount can’t be completed.
In general a little defensiveness here to unreliable external components has only minor cost, and some mitigation seems reasonable.
Opened the first part of the fix in
SG. Yeah, that should be considered a rare case. However, adding some defensive cleanup also SGTM as long as it doesn't introduce too much overhead or complexity. :)
Code change:
[services](services.md) are objects that can be written to and retrieved from the Kubernetes API server. <del> ## Container and Pod Resource Limits </del> <ins> ## Resource Requests and Limits of Pod and Container </ins> Each container of a Pod can optionally specify `spec.container[].resources.limits.cpu` and/or <del> `spec.container[].resources.limits.memory`.  The `spec.container[].resources.requests` field is not currently used and need not be set. Specifying resource limits is optional.  In some clusters, an unset value may be replaced with a default value when a pod is created or updated.  The default value depends on how the cluster is configured. Although limits can only be specified on individual containers, it is convenient to talk about pod resource limits.  A *pod resource limit* for a particular resource type is the sum of the resource limits of that type for each container in the pod, with unset values treated as zero. The following pod has two containers.  Each has a limit of 0.5 core of cpu and 128MiB (2<sup>20</sup> bytes) of memory.  The pod can be said to have a limit of 1 core and 256MiB of </del> <ins> `spec.container[].resources.limits.memory` and/or `spec.container[].resources.requests.cpu` and/or `spec.container[].resources.requests.memory`. Specifying resource requests and/or limits is optional. In some clusters, unset limits or requests may be replaced with default values when a pod is created or updated. The default value depends on how the cluster is configured. If value of requests is not specified, they are set to be equal to limits by default. Please note that resource limits must be greater than or equal to resource requests. Although requests/limits can only be specified on individual containers, it is convenient to talk about pod resource requests/limits.  A *pod resource request/limit* for a particular resource type is the sum of the resource requests/limits of that type for each container in the pod, with unset values treated as zero (or equal to default values in some cluster configurations). The following pod has two containers.  Each has a request of 0.25 core of cpu and 64MiB (2<sup>20</sup> bytes) of memory and a limit of 0.5 core of cpu and 128MiB of memory. The pod can be said to have a request of 0.5 core and 128 MiB of memory and a limit of 1 core and 256MiB of </ins> memory. ```yaml

Related discussion:
text: did which addresses some of them, but turns up a lot more.
Alright, I will do that.