Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: Once we switched over the default runtime to containerd, the slow ci and pull jobs started timing out. The tests, which used to take 13 minutes, jumped to 30 minutes. This increase in test time is only observed in the slow job, so I suspect the slow job is exercising some functionality that may not be exercised as frequently in the other jobs. What you expected to happen: No performance regression How to reproduce it (as minimally and precisely as possible): Run the tests in the slow job with containerd Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
do we know what tests were regressing?
I did some analysis in , but basically, no single test stood out in terms of increasing the time took. I suspect that all the test cases started taking a few seconds longer, and the sum of all of contributed to the increase in job time.
I was mainly eyeballing the graphs, but we probably should get the raw data and do some more detailed analysis on it
do we have scale runs with the [slow] tests?
As an experiment, I changed the runtime back to docker in , and the slow job tests went back down to 13 minutes.
pointed out that may be relevant
/cc karan /cc
pointed out that cri-tools is not used in the kubelet path, so that particular issue may not be the cause. But maybe we can investigate image pulling as a first start. FWIW, there are at least 100+ test cases running in this job that create the same container on the same node multiple times: hostpathtype, multivolume, container restart
I think I found it. All the HostPath Socket tests are consistently never completing.
something might related
The test logs don't seem to show up in the artifacts. Running it manually, I see it hung on the exec during BeforeEach(): However when I do this manually using "kubectl exec" it is fine.
found a work around
i can easily recreate the problem locally using kubetest + local-up- running against containerd.
There are some notes on differences between how dockershim and other CRI impls handle this situation: have you run into this before?
Came up here:
I think what I said in that PR still applies.. I would expect kubectl -it to have a -d detach mode if background operation is the desired mode see docker exec -d option, and clarify what behavior is required of CRI exec sync.. up to 10sec pause for logging return value then return even if the in container exec process is still running? Note the hard coded start ops setting detach to false in docker shim: So it's using synch.. not detached.. but then later docker shim does this: before returning from the ExecSyncInContainer() request.
This seems to be a known issue. If an exec: 1) Spawns a long running background process; 2) Leaves the IO of the background process open after exits; 3) Containerd will wait until that IO to be closed before finishing the exec. It is arguable a bug in containerd, and the behavior of background process spawned by exec was never clearly defined. We've seen several production issues caused by this. Maybe we should: 1) Clearly define and document the behavior, and fix the test; 2) Pass timeout for probe (not related to this issue, but to protect the system). timeout is supported by many runtimes, but kubelet doesn't pass a proper value right now.
Among the callers of HandlerRunner#Run is the following from kubeGenericRuntimeManager#startContainer : We should find a proper value for the timeout.
We have a similar issue with CRI-O. I agree with that we should define this clearly and meanwhile setting a timeout makes sense.
I have opened to fix the tests.

Discussion:
ginkgo.By("Create a pod for further testing") hostBaseDir = path.Join("/tmp", ns) mountBaseDir = "/mnt/test" <del> basePod = f.PodClient().CreateSync(newHostPathTypeTestPod(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate)) </del> <ins> basePod = f.PodClient().CreateSync(newHostPathTypeTestPodWithCommand(map[string]string{}, hostBaseDir, mountBaseDir, &hostPathDirectoryOrCreate, fmt.Sprintf("nc -lU %s", path.Join(mountBaseDir, "asocket")))) </ins> ginkgo.By(fmt.Sprintf("running on node %s", basePod.Spec.NodeName)) targetSocket = path.Join(hostBaseDir, "asocket") <del> ginkgo.By("Create a socket for further testing") _, err := utils.PodExec(f, basePod, fmt.Sprintf("nc -lU %s &", path.Join(mountBaseDir, "asocket"))) framework.ExpectNoError(err) </del> }) ginkgo.It("Should fail on mounting non-existent socket 'does-not-exist-socket' when HostPathType is HostPathSocket", func() {

return pod } <ins> func newHostPathTypeTestPodWithCommand(nodeSelector map[string]string, hostDir, mountDir string, hostPathType *v1.HostPathType, command string) *v1.Pod { pod := &v1.Pod{ ObjectMeta: metav1.ObjectMeta{ GenerateName: "test-hostpath-type-", }, Spec: v1.PodSpec{ NodeSelector:  nodeSelector, RestartPolicy: v1.RestartPolicyNever, Containers: []v1.Container{ { Name:  "host-path-sh-testing", Image: imageutils.GetE2EImage(imageutils.Agnhost), VolumeMounts: []v1.VolumeMount{ { Name:      "host", MountPath: mountDir, ReadOnly:  false, }, }, Command: []string{"sh"}, Args:    []string{"-c", command}, }, }, Volumes: []v1.Volume{ { Name: "host", VolumeSource: v1.VolumeSource{ HostPath: &v1.HostPathVolumeSource{ Path: hostDir, Type: hostPathType, }, }, }, }, }, } return pod } </ins> func verifyPodHostPathTypeFailure(f *framework.Framework, nodeSelector map[string]string, hostDir, pattern string, hostPathType *v1.HostPathType) { pod := newHostPathTypeTestPod(nodeSelector, hostDir, "/mnt/test", hostPathType) ginkgo.By(fmt.Sprintf("Creating pod %s", pod.Name))