Code change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: When I restart containerd.service, kubelet can not connect to it again and cause 100% cpu usage. there are many logs like this: I can show container info via crictl at the meantime and restart kubelet.service can resolve the problem. What you expected to happen: kubelet can reconnect to containerd after containerd restart. How to reproduce it (as minimally and precisely as possible): cpu usage and syslog. Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: armbian(debian on arm64) OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig node
/assign
Reproduce: can reproduce the issue by restarting when is the container runtime another setup where was used used as container runtime, didn't experience the issue by restarting , but restarted the underlying , same issue shown up again and had to restart to resolve the issue. Will continue with the investigation, might need to reconnect to after the socket is broken.
To me it seems there are a couple of problems here. 1. kubelet does not reconnect to the containerd socket. 2. The reconnect is agressive and consumes a lot if cpu. I don't know the cause for 1, but for 2 this might be an indication that the retry interval should be relaxed.
Spent a few hours on the investigation, it's likely to be some internal states being messed up in .
TL;DR: It should be related to gRPC. Hacked the vendored gRPC mod and changed its to something like this: Then CPU only spike for a few seconds after restart then comes back to normal instead of failing the all the time. kubelet passes a customized dialer to the gRPC client. I logged the address during kubelet startup: As you can see, the address of the is at . Before restarts, I can see from log that every time, the gRPC function is using the customized dialer that kubelet passed: However, once got restarted, the address of the dialer is changed: And that function at keeps failing. As I forced to use the original customized dialer, the issue is gone. Without too much knowledge on gRPC, my blind guess is as is passed by reference to gRPC client, some component could modify it during the process lifetime and the change got persisted. Anyway, I'll check with gRPC team on this issue and see if there's already a fix on their side. If so then I can port the fix here.
/cc
Hi, are there any updates on this. I am experiencing what appears to be the same issue. If I restart containerd while the kubelet is running, my kubelet logs get flooded with: Restarting the kubelet appears to be the only way to re-establish the connection with containerd. Version information is:
Hi thanks for the info, I'm still working on the root cause. As I mentioned above, it's something related to gRPC. I collected the object info, this is the object before containerd restart: ! And this is the object after containerd restart: ! There's one thing made me concern is, the connection state is still 2 (i.e. , see ) after the failure. I'm not sure if this is related to the cause but anyway it still should be something related to gRPC. Btw I've followed their to use the latest gRPC client without customized dialer and still see this issue, will still work on the root cause.
We're seeing this now when we update our nodes to 1.19 where our Kata Containers install does a . Our current workaround is to also do , but would be good to get this fixed.
Is this MR relative to the problem?
No I don't think so, I've cherry-picked that change and tried in my environment weeks back, and the issue is still there. I only get around the issue by modifying the gRPC code as I mentioned in the above, I'll continue to check which component modified the function in after thanksgiving.
For the sake of priority. This hit us hard these days too, as security updates for container.d triggered a restart and a ton of nodes flooded the central log management. ;) Thanks for working hard on a fix.
Same here. Two systems were affected by automatic container.d upgrades & restarts. But now I wonder if - generally - "unattended-upgrades" (Ubuntu 18.04) is a wise thing to do? Or should the OS for kubernetes deployments always be updated manually & controlled? I can not find best practices for this in the documentation.
After the recent CVE-2020-15257many users will upgrade and restart containerd. It would be a big problem.
I see, will prioritize on the root cause and fix. At the mean time, you need to restart after you restart .
Had another round of debugging, and made one step towards the root cause. I following snipple to the as I mentioned in previous : When I reproduced the error, I got following information for the dialer function: That leads to cadvisor, which is the only component that uses the in the repo: So I'm convinced that this should be related to and will update the thread once I've made further progress.
EOD updates: issue is indeed caused by cadvisor, reproduced the issue in this . confusing timeout in log should be . root cause might be a bug for using context that already exceeded deadline in cadvisor. To validate that, I need to explore all methods of this : Will work on the fix on cadvisor for the rest of the week.
Fix was made in Copying from to ensure we can capture it on the issue here: I've cut new cAdvisor releases (v0.37.3 and v0.38.6) with this fix cherrypicked. We should get the cAdvisor fix back into kubernetes. We'll need 3 PRs to k/k with following updates: [x] k/k master to update cAdvisor to v0.38.6 -- [x] k/k release-1.20 to update cAdvisor to v0.38.6 -- [x] k/k release-1.19 to update cAdvisor to v0.37.3 --
Thanks for fixing this one, definitely a tricky one to track down.
Thanks for all the patience and team efforts on getting this issue fixed, also credits to on the guides and discussions, won't be able to get it done without your helps!
The fix was applied on master as well as cherrypicked into and releases.
I remember you told me, but I forgot. Does issue exist in 1.18 and 1.17?
I don't believe 1.18 or 1.17 should be affected. I think this issue was introduced with which commit was included in cAdvisor v0.37 release. cAdvisor v0.37 was vendored into k8s 1.19.

Related discussion:
text: github.com/gogo/protobuf v1.3.1 github.com/golang/groupcache v0.0.0-20200121045136-8c9f03a8e57e github.com/golang/mock v1.4.1 <del> github.com/google/cadvisor v0.38.5 </del> <ins> github.com/google/cadvisor v0.38.6 </ins> github.com/google/go-cmp v0.5.2 github.com/google/gofuzz v1.1.0 github.com/google/uuid v1.1.2