Change:
For the cluster upgrade failed. Looking at the failure, it seems to be an inopportune restart of apiserver. The new master comes up, apiserver starts, but then it restarts. The upgrade test hits it during the restart, it looks like. Logs: The reason for the apiserver restart was that kube-apiserver failed its liveness probe The liveness probe for api server is hitting :8080, and indeed it was failing its health check (apiserver log again): I'm guessing this is due to Looking at the next run, I'm hypothesizing that RBAC updates just take too long, and so it's marginal as to whether the bootstrap roles complete in time. /sig api-machinery /kind failing-test cc
/milestone v1.13
I see so many api calls getting clusterroles and clusterrolebingdings. should we run in parallel to accelerate the bootstrap?
I open trying to fix it.
That's great - thank you I do think we should also think about why each call is slow though - running in parallel or just boosting the timeout feels a little bit like cheating, at least until we have justified that this is an intrinsically slow operation. cc (as you know both apimachinery & auth)
I'm seeing a total of 2 seconds to perform all RBAC initialization on initial cluster bring-up, and a total of .25 seconds to verify RBAC initialization on cluster restart. There's nothing inherently slow about the operation. How long is the liveness check configured for in this setup? If many many API calls are being made using the internal client on restart, it could exhaust the assigned QPS and artificially slow down all calls made using it. The apiserver log of calls made during restart (as opposed to initial start) would be helpful in determining if there's a component that is making excessive numbers of API calls.
/priority important-soon
Chatted with directly, and found that while RBAC isn't fast (probably rate-limiting), the're a much bigger delay which is probably more significant: Note the ~ 15 second gap. And this was with other tests as well, e.g. the example above is actually from Going to crank up the log level and see if I get any hints as to what is happening
/assign
GitHub didn't allow me to assign the following users: logicalhan. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I dug into it - the reason for the long delay is that we're waiting for etcd to start. We don't log anything, even at . So maybe we do just increase the wait time :-( An alternative would be to do an initializerPod that waits for etcd, which would be more correct.
I agree with about increasing the wait time for our liveness check (especially considering the fact that we are apparently hitting our liveness failure threshold 75% of the time during out e2e tests).
how did you gather those metrics, by the way?
I'm sure there's a better way, but I ed the logs down with some bash, and then grepped against them. Specifically the file: ( for a count) and to list files without matches
Is this still broken? Trying to figure out if it should be a blocker still, or not. I'm not seeing these test failures.
It’s a bad flake that keeps happening on and off. We can take it off the blocker list, but just find the right owner to investigate and fix for long term. Josh Berkus <:
I discussed this last week with we should increase the livenessProbe delay to avoid the premature kubelet restarts. /assign
GitHub didn't allow me to assign the following users: logicalhan. Note that only and repo collaborators can be assigned. For more information please see <detailsIn response to : I discussed this last week with we should increase the livenessProbe delay to avoid the premature kubelet restarts. Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details

Discussion:
NODE_ACCELERATORS=${NODE_ACCELERATORS:-""} REGISTER_MASTER_KUBELET=${REGISTER_MASTER:-true} KUBE_APISERVER_REQUEST_TIMEOUT=300 <ins> # Increase initial delay for the apiserver liveness probe, to avoid prematurely tearing it down KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC=${KUBE_APISERVER_LIVENESS_PROBE_INITIAL_DELAY_SEC:-45} # Also increase the initial delay for etcd just to be safe ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC=${ETCD_LIVENESS_PROBE_INITIAL_DELAY_SEC:-45} </ins> PREEMPTIBLE_NODE=${PREEMPTIBLE_NODE:-false} PREEMPTIBLE_MASTER=${PREEMPTIBLE_MASTER:-false} KUBE_DELETE_NODES=${KUBE_DELETE_NODES:-true}