Change:
<!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: I have disconnected a node from network for a few minutes. After reconnecting, I keep receiving such error messages from kubelet on the node, even after 15 minutes in reconnected state: What you expected to happen: I expect that after network recovery kubelet reconnects to the Apiserver as before, and after a recovery such timeouts do not occur. How to reproduce it (as minimally and precisely as possible): Just have a node connected to the cluster. Then, disconnect it from the network for 3-4 minutes, then reconnect. Then observe kubelet's logs. Anything else we need to know?: I have strict tcp keepalive settings in place on master and worker nodes, but this should not be the cause. Restarting kubelet solves the issue, the error messages disappear. Environment: Kubernetes version (use ):Cloud provider or hardware configuration: bare metal OS (e.g: ):Kernel (e.g. ):Install tools: kubeadm Network plugin and version (if this is a network-related bug): kube-router Others:
/sig api-machinery
/sig api-machinery
It seems that after some more time, kubelet recovers. Sometimes it takes one hour, sometimes two. Recovery is successful when I receive similar logs: What is the reason that it takes so long for it to recover?
The root cause is , deduplicating against that issue
Debugging, tracing the code, I am not really sure that this is the same issue as the other mentioned here. Error messages show that the client is requesting higher resource versions that is available at the server: , the former is greater by 989 than the latter. I dont know from where this resource version comes from. I've made a very stupid patch on my kubelet/client-go: I am pretty sure that this is not the best solution, but, it seems that it forces a full list every time ListAndWatch is called.
You're correct, sorry about that... I misread the timeout message. The error it is getting is coming from the server, which means kubelet -apiserver connectivity is fine. This message comes from the watch cache, which means the watch cache in the API server is behind where the kubelet's informer was, which is pretty surprising. A few questions: Is there a single apiserver or multi-apiserver cluster? Is there a single etcd member or multiple etcd members? Is there an API server running on the node that was detached/reattached?
cc does this ring any bells?
cc for eyes on watch cache
I don't think I've hit this problem specifically. Since this could happen due to an etcd partition or due to a partition between a apiserver and etcd, the answers questions asked in seem like the appropriate next things to figure out.
Yeah - I agree with Joe. It seems like some network partition issue of the apiserver the node was connected to.
I will be able to answer the questions in detail later, but in short: there are 3 masters set up with kubeadm, haproxy load-balances the queries to the apiservers. The only thing to reproduce the issue is to disconnect the node from the network for a few minutes. Masters or etcd members are not touched, and anyway the cluster seems to be healthy.
Today, it occured on a diffrent cluster. Setup is similar: stacked etcd on master nodes, haproxy load-balances traffic between them. Just was playing with haproxy servers and the HA setup, and after a few restarts of haproxy, one of the nodes entered this state. So during a haproxy restart, all kubelets' connections to apiservers are terminated, but rarely, one of them enters this unhealthy state.
Multi apiserver cluster Stacked etcd members are present. I think In my setup it is not relevant, as all the kubelets are talking to the apiserver through the load-balancer, which picks one of the available apiservers. I am doing a rolling restarts of the nodes, and there were at least one occurence, when even one of the master nodes got stucked in this state. How can I help debugging/fixing this?
How can we proceed with this issue?
got same problem in a old cluster, ( up since 3+years) after updating to kube 1.18.3 , one node kubelet start logging those errors ( ~10 errors/minute with various resources) I've now just restarted kubelet and now seems working again.. Update: while testing I've tried to connect the kubelet of that node to each single master nodes to check if there were some kind of network issue / partitioning.. none of the master servers returned errors ( the kubelet worked fine with all master nodes bypassing the nginx proxy that manage masters HA
I think this problem is getting serious. I am not 100% sure, but today, during node restarts (including masters) coredns was migrated multiple times, and unfortunately, kube-proxy pods had not received the updated situation, thus, dns resolution was not working inside the cluster. After detecting this, a simple restart of all kube-proxy pods resolved our problem. There were no network issues during this, so I suspect we've hit the same bug, now in kube-proxy.
I am trying to dig this. Now, I've written a little go prog which uses the same reflector what kubelet uses. It just starts a watch for CSIDrivers, and I've made client-go to print the last resourceVersion. Starting the program multiple times (i.e. connecting to different masters) produces the following: Right now I dont exactly know what url is being fetched and what arguments are passed in, I still have to figure that out. But now it seems, that different masters return different resouceVersions. Etcd does not report any problems/issues.
Any ideas, suggestions where to look further?
Today I have re-initialized 2 of my 3 master nodes to make sure etcd replicas are not corrupted. Just after the nodes have been joined, the same behavior could be observed: connecting to different masters resulted in different resourceVersions.
Now, it can be seen, that querying different apiservers again return different resourceVersions: 192.168.8.60: is a haproxy load-balancing traffic between 3 masters.
please let me know where to go further. My cluster runs on PI boards, perhaps, not the fastest ones. But if this issue depends on slow hardware, than it can pop up anytime on fast hardware as well.As I wrote, today I reinitialized 2 of my 3 masters to make sure no partitioning occured. Also, please have a look at my comments. Or, if you have a clue that my setup is broken, please let me know!
It seems that it happens to resources where no objects exist at all. Can it be?
Ahh - the fact tht it happens for types with no objects actually made me know what is happening here. Basically what is happening is that when kube-apiserver is starting, watchcache is initialized from "now" (i.e. it will be synced to resource version that was globally last in etcd at the time when it was initialized). However, later it's updated only on changes of resources of those types. So when no objects of that type is being /modified/deleted it will stay the same. So it's kind-of "working-as-implemented" from the kube-apiserver perspective. That can be improved - I'm actually already looking into reusing etcd progress-notify feature to get rid of this behavior ( & ) - I would like to have alpha for that in 1.20 and beta in 1.21 (planning to write down KEP once I'm back from my soon-starting vacation), but that shouldn't be the problem by itself. And it's not what is really biting you. The real problem is that there is a bug in reflector in: It handled the "too old RV" error, but it doesn't handle "too large RV" correctly. What it should be doing instead is to: trigger relist from etcd (to get RV=now") start watching from that RV Watch will work fine because watch will simply wait until events with larger RV appear. I will send out a PR fixing that bug (will try to do it later today).
is out for review
This has been fixed/mitigated in 1.18. Going to backport to 1.18 now. It seems 1.18 is the first version affected (the offending changes were reverted in 1.17 months ago).
This has been mitigated in 1.18 and head. The more proper fix is proposed in Closing this one.
Hello, as i saw, i have the same issue in 1.19.3 version: Our setup is 3 etcd, 2 apiserver bound on those 3 etcd and a haproxy in front of apiservers
If that happens once - that can definitely happen (and that's fine). The bug was that the components were stuck with those errors.
18.0 get same error . restart docker and kubelet return OK. so , should I update the 1.18.0 to 1.18.6+ ?

Discussion:
"io" "math/rand" "reflect" <ins> "strings" </ins> "sync" "time"

return true } } <ins> // Matches the message returned by api server before 1.17.0 if strings.Contains(apierr.Status().Message, "Too large resource version") { return true } </ins> return false }

stopCh := make(chan struct{}) s := NewStore(MetaNamespaceKeyFunc) listCallRVs := []string{} <ins> version := 30 </ins> lw := &testLW{ WatchFunc: func(options metav1.ListOptions) (watch.Interface, error) {

}, ListFunc: func(options metav1.ListOptions) (runtime.Object, error) { listCallRVs = append(listCallRVs, options.ResourceVersion) <ins> resourceVersion := strconv.Itoa(version) </ins> switch options.ResourceVersion { // initial list

err := apierrors.NewTimeoutError("too large resource version", 1) err.ErrStatus.Details.Causes = []metav1.StatusCause{{Message: "Too large resource version"}} return nil, err <ins> // relist after the initial list (covers the error format used in api server before 1.17.0) case "40": err := apierrors.NewTimeoutError("Too large resource version", 1) return nil, err </ins> // relist from etcd after "too large" error case "": <del> return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: "30"}}, nil </del> <ins> version += 10 return &v1.PodList{ListMeta: metav1.ListMeta{ResourceVersion: resourceVersion}}, nil </ins> default: return nil, fmt.Errorf("unexpected List call: %s", options.ResourceVersion) }

// may be synced to a different version and they will never converge. // TODO: We should use etcd progress-notify feature to avoid this behavior but until this is // done we simply try to relist from now to avoid continuous errors on relists. <del> for i := 1; i <= 2; i++ { </del> <ins> for i := 1; i <= 3; i++ { </ins> // relist twice to cover the two variants of TooLargeResourceVersion api errors stopCh = make(chan struct{}) if err := r.ListAndWatch(stopCh); err != nil {

} } <del> expectedRVs := []string{"0", "20", "", "30", ""} </del> <ins> expectedRVs := []string{"0", "20", "", "30", "", "40", ""} </ins> if !reflect.DeepEqual(listCallRVs, expectedRVs) { t.Errorf("Expected series of list calls with resource version of %#v but got: %#v", expectedRVs, listCallRVs) }