Change:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Nodes in status Pods in status What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: AWS OS (e.g. from /etc/os-release): Debian 8 (jessie) Kernel (e.g. ): Install tools: kops Others: Logs:Possible prerequisites: a lot of jobs (1657) that failed (state: Error) because of internal problems Cluster size: 3 masters, 4 worker nodes Workaround: how to get the node backPossible related to
/sig node
It is limitation on grpc. The default value is (i.e. in error message)
do you know if it possible to configure the value with environment variable or maybe in some config file?
Hitting the same bug here (Kubernetes 1.10.2 with docker 17.12.1). The node has 8309 containers, most of them exited, created by failing jobs.
I'll try to change kubelet's ContainerGC so that, when there are more than say 5000 containers it reclaims the oldest ones. Unless someone thinks it's not the right think to do.
can you please try ?
thanks I'll try to reproduce but it will take time (I had to clean the host).
I don't have free infrastructure to install there K8s from some branch and don't have experience to do this at all. I have used only stable releases.
I can already tell that it probably won't solve the problem (with containerd at least):
given what you are seeing won't help but will help. right? (note that default in grpc is )
yes it would work but it's 1GB so we are in a runaway situation here; have 1GB today and you'll need 2GB tomorrow. 16MB is definitely an improvement and your change to 16MB is LGTM (thanks!), but the root problem still remains. The 2 solutions I can imagine are (1) an API change to include paging (or streaming), or (2) make the kubelet ensure there's less than a total number of containers (running+exited). Since the issue also impacts I'm participating in the following upstream issue:
Just discovered the way of the stream seems to be the target:
improves the situation but doesn't completely fix this issue ;-) Should I open another one?
k8s-merge-robot is a bot user; I think you need to open a new issue
it was a humoristic remark ;-) I'm waiting for the improvement in containerd. It seems they're even gonna hide the details about List vs Stream in the client, so we may just have to bump de client version at some point here.
Exactly! We just downed a docker swarm mode cluster with mere 80 services with 1 container each.
run into this issue as well, dead container did not remove by gc, this functionality seems deprecated in recent k8s release, no sure why. see
I met the problem for k8s 1.13.2 .Do you have any solutions except clear dead container manually?
Really? There's an absolute universe more information in 1GB that 4MB,, just as it made sense to not make the limit 4096. Although the above is true, can we agree there is huge difference, in human terms, between 4MB and say 1GB. Pretty please change the limit to a round 1GB or something that is configurable at run-time by the service publisher or as a connection option. Maybe an option flag that waves this limit at runtime? This is a scary issue for us b/c now we have to write a segmenter/desegmenter in our client? Fear. Please let consumers of pb tech not have to do this. Everyone will just have to write their own segmenter/desegmenter hurting the pb's rep as the one stop shop for interoperabilty.
streaming is native in protobuf...
Sure... but we're hitting this limit and we're already in a stream call. :
On Jul 21, 2019 17:36, drew-512 <: Sure... but we're hitting this limit and we're already in a stream call. :Sounds weird, you're talking about the limit of the Kubernetes' kubelet gRPC client talking to the CRI runtime? We never got this bug again and it's an old one.
While doing some performance testing with AWS EKS, I ran into a situation where pods would have "CreateContainerError". These were created in a non-default namespace on CentOS 7.4 hosts. Within this cluster - remember it is used for performance testing - there were over 20,000 deployments, all in a failed state. Worker nodes would have entries like: These pods were in a namespace other than default. However, I was able to successfully scale coredns as desired without a problem. So I learned this problem was limited to a namespace. I deleted the namespace and recreated it and I could successfully deploy pods. docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?
On Jul 22, 2019 18:31, Dave Augustus <:docker --version Docker version 18.09.8, build kubelet --version Kubernetes v1.12.7 So is there a way of dealing with this error other than deleting/creating the namespace? Some cleanup process? or parameter adjustment?AFAIK using docker does not go through the same interface as containerd. You should probably be able to recover the situation by removing containers (ie: docker system prune).
For anyone operating a k8s cluster, running into this error and not being sure what the actual problem is... If you run this , you'll see that a message describing all containers currently known to docker is bigger than the limit the Kubelet has set for gRPC messages (16MB at the time of writing). This either means you have a LOT of (possibly dead) containers, or there's a LOT of metadata attached to your containers. This metadata probably comes in the form of custom labels attached to the container image by whatever tool you're using to create the container images. Check out for a couple of your containers and the issue should become apparent.
Thanks your comment was really helpful!! However, is there any systematic solution to resolve this? rather than on such node or remove dead containers manually? I mean, is there any chance to increase limit of gRPC message size?
My case was caused by too many tekton dead pod (over 3000 dead containers). I will try to add to the kubelet to solve the issue
Thanks for sharing! :+1:
My case was caused by too many tekton dead pod (over 5000 dead containers). the kubelet get the message to solve this issue. first , add to kubelet comand line . second . restart kubelet third. recheck and problem is solved. thanks to
I'm glad it helped. In my case, after adding , it still happened once after. So I have to add a docker-prune service for double insurance. Just for reference /etc/systemd/system/docker-prune.service /etc/systemd/system/docker- It will run once a week on Sunday Using systemctl enable and start both. check the status with .
The solution here for me, install nerdctl, then: Once containers are pruned, you need to restart containerd for the node to show ready again.

Discussion:
"k8s.io/kubernetes/pkg/util/interrupt" ) <ins> // maxMsgSize use 8MB as the default message size limit. // grpc library default is 4MB const maxMsgSize = 1024 * 1024 * 8 </ins> // DockerServer is the grpc server of dockershim. type DockerServer struct { // endpoint is the endpoint to serve on.

return fmt.Errorf("failed to listen on %q: %v", s.endpoint, err) } // Create the grpc server and register runtime and image services. <del> s.server = grpc.NewServer() </del> <ins> s.server = grpc.NewServer( grpc.MaxRecvMsgSize(maxMsgSize), grpc.MaxSendMsgSize(maxMsgSize), ) </ins> runtimeapi.RegisterRuntimeServiceServer(s.server, s.service) runtimeapi.RegisterImageServiceServer(s.server, s.service) go func() {