Change:
Which jobs are flaking: pull-kubernetes-e2e-gce ci-cos-containerd-e2e-ubuntu-gce ci-kubernetes-e2e-kops-gce Changes needed to get the fix into official containerd releases: [x] runc race condition fixed - [x] runc release - [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - As I understand it, this is what we need to get the fix into our CI environments: For kind: [x] kind containerd build with updated runc - [x] kind containerd-nightly release include patch - [x] kind containerd-nightly release - [x] kind pick up new containerd release - For GCE / kube-up: [x] [x] cri bump of runc - [x] containerd bump of runc/cri - [x] containerd 1.2.x release - [x] containerd 1.3.x release - [x] COS image update with new runc version - Which test(s) are flaking: [sig-storage] ConfigMap should be consumable from pods in volume as non-root with FSGroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected downwardAPI should provide podname as non-root with fsgroup [LinuxOnly] [NodeFeature:FSGroup] [sig-storage] Projected configMap should be consumable from pods in volume with mappings as non-root [NodeConformance] [Conformance] [sig-storage] EmptyDir volumes should support (root,0644,tmpfs) [LinuxOnly] [NodeConformance] [Conformance] [sig-storage] In-tree Volumes [Driver: hostPathSymlink] [Testpattern: Inline-volume (default fs)] subPath should support file as subpath [LinuxOnly] [sig-storage] In-tree Volumes [Driver: local][LocalVolumeType: blockfs] [Testpattern: Pre-provisioned PV (default fs)] subPath should support readOnly file specified in the volumeMount [LinuxOnly] Testgrid link: Reason for failure: runc race condition fixed in opencontainers/runc Anything else we need to know: Test containers appear to be exiting with a CRI error? /sig node for CRI expertise /sig storage for test ownership
From in
I think we have a general failure to start containers reliably. Almost every instance of the "In-tree volume" test flakes referenced from contained the pod status error of "OCI runtime start failed: container process is already dead: unknown"
seen in [] Docker Containers should be able to override the image's default arguments (docker cmd) [NodeConformance] [Conformance] in
also seen in containerd CI runs -
This seems to be a race condition in runc. It only happened once in 2 weeks for containerd. In the runc code, when starting a container, will be called: There are 2 cases: : returns and throws out error, when 1) exec fifo is opened; 2) after 100 milliseconds and the init process is zombie or gone. : returns when the exec fifo is opened. If I understand the code correctly, I can see a potential race condition: 1) opens the fifo, but hasn't informed yet; 2) Container process starts running after the exec fifo is open; 3) Container process finishes; 4) checks process after 100 milliseconds, and found the process is already gone; 5) The error is returned unexpectedly. At least for the containerd failure, the test is a short running container.
thanks for digging. at first glance, it looks more frequent than that in some of the jobs - ci-cos-containerd-e2e-ubuntu-gce - Failed in 5% (18/348) of builds: 12/17/2019, 10:14:46 AM 12/15/2019, 2:28:43 PM 12/14/2019, 7:09:32 PM 12/14/2019, 12:02:33 PM 12/14/2019, 4:55:35 AM 12/13/2019, 2:42:28 PM 12/13/2019, 12:40:08 PM 12/13/2019, 5:33:22 AM 12/12/2019, 8:24:21 PM 12/10/2019, 2:20:14 AM 12/9/2019, 6:01:32 AM 12/8/2019, 12:44:23 PM 12/7/2019, 8:27:44 PM 12/6/2019, 9:04:54 PM 12/6/2019, 1:58:22 PM 12/5/2019, 12:33:37 PM 12/4/2019, 4:01:54 AM 12/3/2019, 7:42:04 AM
/remove-sig storage
opened
cc
fix in progress at
the runc race condition fix has been merged at I think the next steps are getting a runc release cut and propagating that fix into the container runtimes we use in Kube CI (docker and containerd). Who on the sig-node side would be the right people to drive those?
For the GCE tests, we'd need a new COS image once the new runc release is available.
once a runc release is available, we'll need to update our e2e images /assign for COS image updates /assign for kind updates
as soon as there's a runc release we'll roll it out in kind (preferably if containerd 1.3.X can pick it up officially... we already have infra building that), tempted to go ahead and grab a runc binary at some commit ...
looks like runc rc10 is blocked on a fix for (which should have a PR open tomorrow per )
We will have to request containerd folks bump opencontainers/runc here - this is a nasty bug, hope we get a runc release quickly (cc
and then request docker folks after that -
Also affects (I guess)
Still waiting for but there's recent activity :crossed_fingers: On the KIND side: When this goes in and there is a runc release, next up after containerd bumping to it we'll (I'll) ensure kind picks up the new containerd release and builds new images with it. Right now we consume the 1.3.X / release-1.3 branch, but we are good to upgrade to HEAD if necessary.
On the GCE side: I'm still not sure who handles COS etc. currently, but someone from node on our end (=Google) will need to get this ported into COS and into our CI.. I'm going to poke as a starting point
runc rc10 is cut. containerd and docker are up next?
On Thu, Jan 23, 2020, 19:10 Jordan Liggitt <:
edit: Hoisted checklist of remaining work to the issue description
KIND won't actually need a containerd release, just a build of the commit with updated runc. Additionally though: per discussion we've noted that the runc bump is mostly information to downstream packaging of containerd, there are no code changes. So we're tentatively forging ahead with bumping runc in our "nightly" build ahead of that landing in containerd.
looks clean after merged. I verified the only flakes occurring after that was merged were on past branches still using the older COS versions. Next week, we can update the non-test kube-up config file to use the fixed version of COS as well if signal still looks good.

Discussion:
# you are updating the os image versions, update this variable. # Also please update corresponding image for node e2e at: # https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/jenkins/image-config.yaml <del> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-114-0} </del> <ins> GCI_VERSION=${KUBE_GCI_VERSION:-cos-77-12371-175-0} </ins> MASTER_IMAGE=${KUBE_GCE_MASTER_IMAGE:-} MASTER_IMAGE_PROJECT=${KUBE_GCE_MASTER_PROJECT:-cos-cloud} NODE_IMAGE=${KUBE_GCE_NODE_IMAGE:-${GCI_VERSION}}