Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: In our stress testing, we found the deadlock issue in PLEG relist() for healthy checking and Kubelet syncLoop() The goroutine stack trace is as following We did some investigation, and found the following issue. The main loop for event processing in Kubelet will check if there are some runtimeErrors: if yes, wait until runtime back to normal; if no, it will call syncLoopIteration to read and process events. And the runtime healthChecks depends on the But in the relist() impl, it will send the pod lifecycle event to eventChannel which may introduce deadlock when eventChannel is full. And in the above code, the eventChannel will not be consumed if the runtime is not healthy. In some situation, the eventChannel may full, and the relist() will hang in sending new pod lifecycle event. And the runtime status will not be able to recovered. It is related to issues What you expected to happen: No such issue How to reproduce it (as minimally and precisely as possible): It is not easy to reproduce, it happens after few days stress testing (scaling the deployment repeatedly with high load). Anything else we need to know?: Environment: Kubernetes version (use ): 1.11.5 Cloud provider or hardware configuration: Bare metal OS (e.g. from /etc/os-release): CentOS 7.4 Kernel (e.g. ): Linux bjc-qtt-k8s-node-010 3.10.0-693.2.2.el7.x8664 SMP Tue Sep 12 22:26:13 UTC 2017 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig node
Sure, thank you We had some internal fix to break the deadlock, but I am wondering if there are good fix for such kind of issue.
When pleg do relist, can we invoke to judge runtime is ok, if not, not do relist
I submit a PR
Thanks for finding the issue! The issue description makes sense to me. I think PLEG is a core component, we should try our best to unblock it. The event channel becoming full seems more like an issue in the event consumer side, which shouldn't block PLEG. One solution is to queue the events in PLEG and send later, but that doesn't fundamentally solve the problem, if we don't set a limit on the event number we queue, it seems like a potential memory leak to me; if we set a limit, we may eventually hit this problem again. Another solution is to add a retry for sending event in PLEG, and set a bit in PLEG in between the retry which tells that PLEG is not blocking, it is trying to send out the event. And in the function of PLEG, it returns healthy if the bit is set. Actually, even if we fail to send the event and discard it, since we have already updated the pod cache, the periodical pod resync should eventually pick it up. This may delay pod sync for up to . But if the channel is full, it means that either 1) the syncloop is blocked for some reason, or 2) the node is under heavy loads. In both cases, even the event is successfully sent, the pod sync will likely be delayed anyhow. We can: 1) Enlarge the event channel size to reduce the chance that the channel becoming full. Currently it is arbitrarily set as . 2) Discard the event in PLEG if the channel is already full. In the 3 options above, I'm fine with either 2 or 3. Option 2 doesn't have the 1min pod sync delay, but a bit ad-hoc; option 3 is more generic, but may delay pod sync. There might be other options. /cc Any thoughts?
If PLEG is already blocked by a full event channel, this doesn't help right?
cc/ This looks like the one production issue you mentioned to me a while back.
of options enumerated, option 3 was preferred for me.
Thanks for reporting the issue! what was the load on the node that triggered this condition? If the node wasn't function at all, and the events slowly built up over time, bumping the size of the channel won't help. This should work, and doesn't complicate the logic in PLEG more, so it's my preferred option too. We need some unit tests, and ideally, a stress test, as mentioned in the sig-node meeting.
Agree, giving a proper size for event channel is tricky as the scale also grows with time. We will ship fix as option 3 () for 1.14. Thanks for summarizing all these up as sig meeting! /assign
If possible, could you share in how large the stress is? It helps understand the issue and kubelet stress handling more. :)
Sure, our product team ( ) will input the stress test data after removing sensitive information in follow up comments.
Thanks for reply. Following is our testcase to reproduce this issue: The test cluster has 5 node on aliyun ECS(like EC2). Three 4C8G machine as master node, Two 96C384G as worker node. Firstly, We create 2 deployment running on the worker node. One deployment is http server pods. Another is siege pods to produce network stress to http server deployment. Those stress test deployments produce 50 CPU Core load on worker node. Then we create a job frequently create/delete 20 pod every 5 seconds. After about 1 day of stess test, The worker node will become NotReady. Dig the pprof goroutine stacktrace found the channel block in .
Yeah, I can see how frequent container state change can cause this issue. The number is very useful, thanks a lot!
cloud you please help to share how to dump the goroutines stack trace of kubelet like what you pasted in your comment. thanks a lot.
This issue is fixed by
using pprof is a answer to my question thank you anyway.

Related discussion:
text: verifyEvents(t, expected, actual) } <ins> // TestEventChannelFull test when channel is full, the events will be discard. func TestEventChannelFull(t *testing.T) { testPleg := newTestGenericPLEGWithChannelSize(4) pleg, runtime := testPleg.pleg, testPleg.runtime ch := pleg.Watch() // The first relist should send a PodSync event to each pod. runtime.AllPodList = []*containertest.FakePod{ {Pod: &kubecontainer.Pod{ ID: "1234", Containers: []*kubecontainer.Container{ createTestContainer("c1", kubecontainer.ContainerStateExited), createTestContainer("c2", kubecontainer.ContainerStateRunning), createTestContainer("c3", kubecontainer.ContainerStateUnknown), }, }}, {Pod: &kubecontainer.Pod{ ID: "4567", Containers: []*kubecontainer.Container{ createTestContainer("c1", kubecontainer.ContainerStateExited), }, }}, } pleg.relist() // Report every running/exited container if we see them for the first time. expected := []*PodLifecycleEvent{ {ID: "1234", Type: ContainerStarted, Data: "c2"}, {ID: "4567", Type: ContainerDied, Data: "c1"}, {ID: "1234", Type: ContainerDied, Data: "c1"}, } actual := getEventsFromChannel(ch) verifyEvents(t, expected, actual) runtime.AllPodList = []*containertest.FakePod{ {Pod: &kubecontainer.Pod{ ID: "1234", Containers: []*kubecontainer.Container{ createTestContainer("c2", kubecontainer.ContainerStateExited), createTestContainer("c3", kubecontainer.ContainerStateRunning), }, }}, {Pod: &kubecontainer.Pod{ ID: "4567", Containers: []*kubecontainer.Container{ createTestContainer("c4", kubecontainer.ContainerStateRunning), }, }}, } pleg.relist() // event channel is full, discard events expected = []*PodLifecycleEvent{ {ID: "1234", Type: ContainerRemoved, Data: "c1"}, {ID: "1234", Type: ContainerDied, Data: "c2"}, {ID: "1234", Type: ContainerStarted, Data: "c3"}, {ID: "4567", Type: ContainerRemoved, Data: "c1"}, } actual = getEventsFromChannel(ch) verifyEvents(t, expected, actual) } </ins> func TestDetectingContainerDeaths(t *testing.T) { // Vary the number of relists after the container started and before the // container died to account for the changes in pleg's internal states.