Change:
<!-- Please only use this template for submitting reports about flaky tests or jobs (pass or fail with no underlying change in code) in Kubernetes CI --Which jobs are flaking: pull-kubernetes-e2e-gce ci-kubernetes-e2e-gci-gce Which test(s) are flaking: [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] [sig-storage] In-tree Volumes [Driver: emptydir] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link] [Testpattern: Pre-provisioned PV (default fs)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Inline-volume (default fs)] volumes should store data [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory [sig-storage] In-tree Volumes [Driver: gluster] [Testpattern: Inline-volume (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: hostPath] [Testpattern: Inline-volume (default fs)] subPath should be able to unmount after the subpath directory is deleted [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: tmpfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing directories when readOnly specified in the volumeSource [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: dir-link-bindmounted] [Testpattern: Pre-provisioned PV (default fs)] subPath should support existing single file [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: block] [Testpattern: Pre-provisioned PV (ext3)] volumes should allow exec of files on the volume [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (default fs)(allowExpansion)] volume-expand Verify if offline PVC expansion works [sig-storage] In-tree Volumes [Driver: gcepd] [Testpattern: Dynamic PV (ext3)] volumes should store data Testgrid link: Reason for failure: Failure fetching metrics: Anything else we need to know: This appears to have restarted abruptly around 2/11-2/12 /sig storage /priority important-soon
xref I think we have a general failure to start containers reliably. Almost every instance of these test failures contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
after sweeping a bunch of the in-tree test failures, almost all seemed to have as a root cause, triggered by very short-lived containers used to check file content or permissions. The only test I saw failing that didn't seem to fit that was: [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support non-existent path [sig-storage] In-tree Volumes [Driver: nfs] [Testpattern: Dynamic PV (default fs)] subPath should support existing directory examples: I'd recommend repurposing this issue to address that specifically and leave the OCI issue for the short-lived container issue
Here's a PR for context: So, I think this PR is another example of flakes. Here are some links to previous runs: and same message: This prow: has the above message and a few more, all A lot of in-tree volume issues, but there were some persistent volume ones too. But, that showed up once so far from what I've seen.
The "error getting c-m metrics" failures appear to have just restarted:
/priority critical-urgent bumping priority since we're seeing significant numbers of failures that just started in the last two days
this is failing many test runs per day, do you have updated status on this?
I haven't had chance to fully debug this yet. looked into this little bit and he suspected that these tests started to flake same time around was merged. I should be able to take a closer look tomorrow.
This occurrence fails because it's trying to get the metrics from the kube-controller-manager pod but it is not ready until one minute later if we grep in the kube-apiserver for the controller pod name we can observe that after the pod is up it starts to get 200 instead of 404
keeping open until shows this is resolved
seen again in
this time the container is running but still it can't find it
the errors comes from the request to the apiserver on How can we check that the pod is found and that the apiserver found it?
the first successful get of /metrics logged by the controller manager in that run is at: We're also not checking the error in the once.Do to make sure the pod was running, and once running, we should probably then also check the metrics path on the pod. Maybe something like this:
still happening: message logging missed using the correct errors, so the messages aren't helpful:
what's this stack trace in the apiserver? is this related? a 503 shouldn't cause that, right?
again it seems is not really waiting because it fails just 200 ms after the INFO msg, I think is because it checks if the pod is running, but the message is that the pod doesn't really exist. The fact that there are 9 tests failing with the same error and we are using has to be related to the way Ginkgo use parallel.
It turns out that the method used to wait for the controller manager pod fails if the pod doesn't exist
Ah, so first we should wait for the pod to exist, then be ready, then return metrics
This start to look promising
yes, looks good. thanks for all the work /close
Closing this issue. <detailsIn response to : yes, looks good. thanks for all the work Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Fixed by

Discussion:
"//staging/src/k8s.io/client-go/kubernetes:go_default_library", "//staging/src/k8s.io/component-base/metrics/testutil:go_default_library", "//test/e2e/framework/log:go_default_library", <ins> "//test/e2e/framework/pod:go_default_library", </ins> "//test/e2e/perftype:go_default_library", "//test/e2e/system:go_default_library", "//vendor/k8s.io/klog:go_default_library",

import ( "context" "fmt" <ins> "sync" </ins> metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/fields" clientset "k8s.io/client-go/kubernetes" api "k8s.io/kubernetes/pkg/apis/core" "k8s.io/kubernetes/pkg/master/ports" <ins> e2epod "k8s.io/kubernetes/test/e2e/framework/pod" </ins> "k8s.io/kubernetes/test/e2e/system" "k8s.io/klog"

// Grabber provides functions which grab metrics from components type Grabber struct { <del> client                    clientset.Interface externalClient            clientset.Interface grabFromAPIServer         bool grabFromControllerManager bool grabFromKubelets          bool grabFromScheduler         bool grabFromClusterAutoscaler bool masterName                string registeredMaster          bool </del> <ins> client                            clientset.Interface externalClient                    clientset.Interface grabFromAPIServer                 bool grabFromControllerManager         bool grabFromKubelets                  bool grabFromScheduler                 bool grabFromClusterAutoscaler         bool masterName                        string registeredMaster                  bool waitForControllerManagerReadyOnce sync.Once </ins> } // NewMetricsGrabber returns new metrics which are initialized.

if !g.registeredMaster { return ControllerManagerMetrics{}, fmt.Errorf("Master's Kubelet is not registered. Skipping ControllerManager's metrics gathering") } <del> output, err := g.getMetricsFromPod(g.client, fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName), metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </del> <ins> podName := fmt.Sprintf("%v-%v", "kube-controller-manager", g.masterName) g.waitForControllerManagerReadyOnce.Do(func() { e2epod.WaitForPodNameRunningInNamespace(g.client, podName, metav1.NamespaceSystem) }) output, err := g.getMetricsFromPod(g.client, podName, metav1.NamespaceSystem, ports.InsecureKubeControllerManagerPort) </ins> if err != nil { return ControllerManagerMetrics{}, err }