Change:
When running conformance testing on a cluster with 8 nodes schedulable for jobs, the conformance test for rolling updates on daemon sets fails due to timeout. Test name: Test code is here: It utilizes a hardcoded timeout limit of 5m. What happened: The logs seem to indicate nothing wrong other than the time it takes to complete the rolling restart. The rolling restart timeout is hardcoded at 5m which doesn't seem to be enough time for large enough clusters. What you expected to happen: I would expect the timeout to either be sliding or for the check to be kicking the can; ensuring that the update is proceeding by at least one node every minute or two. How to reproduce it (as minimally and precisely as possible): Requires a cluster with at least 8 schedulable nodes; more would make the timeout more certain. Anything else we need to know?: I was trying to see what k8s was testing itself with and it seems (according to this log) that the test is run against a cluster with 4 schedulable nodes and takes ~3m which would line up with what I'm seeing (8 nodes for ~6m, 1 more than given by the hardcoded deadline) Seems like there are a few options: use a "kick the can" strategy to ensure the timeout scales use a timeout calculated from the number of schedulable nodes (mitigation) increase the max number of down nodes so the update can occur faster Environment: Kubernetes version (use ): From test logs (cluster no longer available): /sig testing /kind bug
If there is agreement on the issue, I'm happy to implement the fix for this. I couldn't find anyone else really complaining about this timeout though so I wanted to be sure to get a sanity check first though.
Can I get someone to confirm/debunk my analysis above?
I am running into this as well. It seems like the first image () that is being used in the test takes around 30-40 seconds to be terminated. The test consistently hits the 5 minute timeout on a 10 node cluster, given that rolling all the pods takes longer than 300 seconds. The binary [sleeps for 60 seconds][0] after getting the SIGTERM signal, which I believe would cause the kubelet to force-kill the process after the default 30 second grace period. Not sure why that sleep is there. [0]:
i can repro this in kubeadm-dind with It's a simple timeout issue. The behavior has changed between 1.11 and 1.12 in 1.11 this test takes less than 5 minutes for 10 nodes in 1.12 the test is now failing consistently
I don't know if I'm going to have enough bandwidth on this right now to dig into this. It's likely the shift from controller<scheduler on daemonsets. I don't know if/when that landed. It seems kind of strange that the test takes that long TBH.
I think the issue is in the test image that we are using, and not necessarily in the controller/scheduler. Seems like there was a sleep introduced to the serve-hostname process as part of this PR: I am wondering if it was mistakenly left in there. If the sleep weren't there, the workload should exit immediately, allowing the rolling upgrade of the daemonset pods to complete within the allocated timeout.
/area conformance
I am working on a PR that removes the sleep from . In the same PR, I am planning to bump the image version, and update all references I can find to the image.
Thanks for all the eyes looking at this, I appreciate it. Removing the sleep will obviously speed things up and probably pass in these use cases. It still seems like something should be done so that the timeout scales though; if we have 1k node clusters we'd probably still have a problem, yes?
/assign We're going to need an image push update for conformance regression. WRT timeout scaling... yes, but let's base it on some data and experiments.
The sleep in there was over a year ago. What changed RECENTLY? Curiously there is a serve-hostname v1.4 tagged on already, but I can;t say where it came from, since VERSION in k/k/test/images/serve-hostname says 1.2 ... That sleep was to test graceful termination, I think. removing it may have side-effects. Can we make the test runs set ? Or maybe we make this termination-sleep a flag? If we change it at all, we should bump the version to 1.5 and supercede all others.
Thanks The most recent, related change I could happened 6 months ago: (PR ). With this change, the conformance test started using the 1.1 version of the image, which includes the sleep. I think it'd be great to know if and how this sleep is being used in tests. also suggested (in ) using a flag to control the sleep behavior, so perhaps has more details about how the sleep is used. With that said, it seems like we could address the immediate issue by setting the termination grace period to 1 or 2 seconds in the DaemonSet rolling upgrade test. et al. - Would love your thoughts on whether this is an acceptable fix.
cc
cc
Agree this needs fixing Will defer to DaemonSet owners on correct route to fix, but I like suggestion. Perhaps we can just use a smaller sleep.. I see in the linked PR that we're bumping to 1.3 despite the mystery 1.4 image, we should bump to 1.5 per other point
What images we use for this test shouldn't matter, as we only care about whether the images is updated. So if serve-hostname is causing us problem we can just use some other images instead. We could also increase the timeout based on the number of nodes for DaemonSet rolling update test.
We have 45 nodes and we are unable to get this test to pass as well due to timeouts. Changing the image might help, but maybe adjusting the timeout based on node count is a solution as well.
totally. suggested that as well and we have it to the PR

Discussion:
f = framework.NewDefaultFramework("daemonsets") <del> image := framework.ServeHostnameImage </del> <ins> image := NginxImage </ins> dsName := "daemon-set" var ns string

ds, err = c.AppsV1().DaemonSets(ns).Patch(dsName, types.StrategicMergePatchType, []byte(patch)) Expect(err).NotTo(HaveOccurred()) <ins> // Time to complete the rolling upgrade is proportional to the number of nodes in the cluster. // Get the number of nodes, and set the timeout appropriately. nodes, err := c.CoreV1().Nodes().List(metav1.ListOptions{}) Expect(err).NotTo(HaveOccurred()) nodeCount := len(nodes.Items) retryTimeout := dsRetryTimeout + time.Duration(nodeCount*30)*time.Second </ins> By("Check that daemon pods images are updated.") <del> err = wait.PollImmediate(dsRetryPeriod, dsRetryTimeout, checkDaemonPodsImageAndAvailability(c, ds, RedisImage, 1)) </del> <ins> err = wait.PollImmediate(dsRetryPeriod, retryTimeout, checkDaemonPodsImageAndAvailability(c, ds, RedisImage, 1)) </ins> Expect(err).NotTo(HaveOccurred()) By("Check that daemon pods are still running on every node of the cluster.")