Code change:
After reading up on some corner cases with etcd client behavior (connections, streams, etc.) I came across the following option that has been introduced in etcd a while ago: From the official etcd docs on client behavior: Source: I do not see this option being used in the API server when is . Is there code in the API server that deals with these cases (etcd/API server node paritioned) and disconnects clients (REST, SDK) appropriately? Or are we at risk of having dangling (hanging) consumer-side client watches due to not using this option in the etcd client? Related discussions: /sig api-machinery cc/
was this something that was considered already?
We should be using for watches. (see also the ). It should just need to be . It would also be valuable to confirm that the behavior works as expected both when establishing watches and when a etcd member is partitioned after the watch has been established.
If anyone wants to own fixing this please self assign, else feel free to assign to me.
For this method: It seems we should do:
Thanks! This is a good question. I am not exactly sure what is the semantics on kubernetes watch. It is correct that without , etcd server will not cancel watch stream when itself is isolated from the cluster. However, when this happens, other requests to etcd (including health checks on etcd) will fail. So the control plane is already in a bad state and is likely to be replaced / repaired either automatically or manually. IIUC, this means apiserver will restart, the its clients will restart their watches.
Oh I just saw Joe's earlier comment. If using aligns with k8s watch semantics, we should use it:)
i'd support moving to 3.4.5 to pick this up!
I am happy to contribute but want to do some more testing and validation on the current behavior of and with the proposed fix. Please give me some days.
Quick update: after some quick tests it seems that does not solve the issue with an active after the connected etcd endpoint becomes partitioned. I think we have the following scenarios to consider: 1) Single node API server/etcd 2) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is colocated with the etcd nodes (e.g. one API server and one etcd instance on one node) 3) Quorum-based cluster (e.g. 3 nodes) where K8s control plane is not colocated with the etcd endpoints (separate nodes for K8s control plane and etcd nodes) We can rule out (1) because technically a partition with just one node in the cluster (K8s control plane and etcd) cannot happen. (3) is mostly addressed with existing gRPC timeouts and keep-alives so when an API server connected to an etcd node that gets partitioned this API server will reconnect to other reachable etcd nodes (endpoints). However, this is only true if the API server itself is not within that partition (e.g. multi-zone deployment where a whole zone becomes partitioned; read on for my observations). (2) is the interesting one: an API server is connected to an etcd node and both become partitioned from the majority. But the API server has active watches against this etcd node and serves these (Kubernetes) watches to Kubernetes clients (controllers, etc.). I was expecting that the gets evaluated even after establishing the . But it seems my assumption is not correct, i.e. it is only evaluated (checked) when is called. At least I did not see an existing being dropped after partitioning the nodes from the quorum (I did check the underlying gRPC transport status and it remained in stayed for as long as the partition was not resolved. So what does that mean for the issue raised here? One could argue that if more than just one etcd node is partitioned, i.e. also the API server and perhaps other Kubernetes controllers running in that partitioned environment (e.g. a zone), it makes no difference using because: the clients (custom controllers, etc.) are also partitioned and thus cannot connect to any other API server anyways; would be nice to return an error here instead of hanging but the system won't self-heal during the partition even if the clients (custom controllers) would not be in that partition, the corresponding API server could not be reached anymore and client-go session handling should address this (e.g. reconnect to another API server) you write: I did not see an existing watch being dropped, e.g. using this simple code: Once I partition the etcd node including this client it will not drop the . Looking at the definition of it seems that this is performed only during establishing the but not reevaluated as long as the client has a successful connection to the etcd node (e.g. node is partitioned but not dead). Since I am by no means an expert on the etcd semantics here, I might be doing something wrong. It would be great to get some clarification whether this issue (I raised, I know :) ...) needs to be addressed, documented, or can actually be addressed at all with the current implementation of .
Please see analysis above. should this work when the etcd + apiserver are on the same node or not?
Thanks for the careful analysis My take on this is: For case (2) where the apiserver and etcd are co-located (and the apiserver only talks to that etcd), I don't see any reason why dropping the watch would help. In this type of setup, when etcd becomes partitioned, the co-located apiserver becomes unavailable and apiserver clients should route away from that apiserver entirely until the partition is healed. Even if the apiserver were to drop the watch, it is still unavailable. For case (3) the behavior is already better than I expected. optimizes the case slightly by ensuring the initial watch connection is against a non-partitioned etcd, so it's still worth enabling, but not urgently needed since failover is already in place. We should document in etcd that only applies to establishing the initial watch connection, the documentation is currently misleading.
It is correct that without WithRequireLeader, etcd server will not cancel watch stream when itself is isolated from the cluster This is correct. Edit: Sorry I missread the test. IIUC, this is not correct. After receiving the watch request with , etcd server will continue to monitor its leader for this watch stream. When there is leader lost observed for "maxNoLeaderCnt election timeouts", server will cancel the watch stream.
Thanks that means the documentation is correct then. So while we haven't found any pressing issues that would fix, it does do what it should do, enabling it in the kube-apiserver won't make anything worse, and might make failover slightly faster?
Make sense to me.
BTW, I just did a quick test. an etcd cluster of 3 members: A, B and C. member B and C some time, the etcdctl command prints
Sorry for my delayed reply, wanted to do some more tests before replying. I can confirm this scenario. However, the tests that I did and which caught my attention because it was not what I expected from reading the doc string was to not shut down the peers but isolate one peer, i.e. simulate a partition which technically is also a "lost leader" scenario. In this case I saw the watches not being closed (neither for nor programmatically). Not sure if this is a bug or by design but I think it's more likely to have partitions than loosing the whole cluster where a majority of nodes is terminated/crashed (stop-failure). If you think it's worth and not expected behavior I am happy to open an issue in the etcd repo and provide more details. Even though in etcd makes it sound should address the partition scenario. Note: I'm currently reviewing the network partition tests in etcd which state that the results I'm seeing are not expected. Update: looks like I'm experiencing what was reported a while ago in but it's unclear to me whether it's fixed, unfixable or what the workaround (from K8s perspective) would be.
I'm in favor of you opening a new against etcd for the watch not being closed in the lost leader scenario (and linking to this discussion). If it turns out it is a duplicate we can always mark as such.
has some detailed watch behaviors.
FYI. There was a bug in . The fix was backported to = v3.3.19 and v3.4.5
Thank you and for your feedback. After reviewing the docs and I think the issue with and how to deal with partitions is still open (see TODO below). The scenario where clients (Watchers or any client indirectly consuming from an etcd ) would hang (forever?) is when the client gets partitioned with the corresponding partitioned etcd node OR when a whole zone/dc is partitioned. Since etcd picks safety (consistency) over liveness (availability) in this case at least not data corruption occurs but the overall system would be stuck when using watches instead of PUT/GET RPCs with timeouts. It would be ideal to throw a useful error, e.g. "lost leader", instead of hanging in "gRPC TRANSIENT_ERROR" state (which is what I'm seeing). Until this is resolved in etcd, perhaps the docs could be updated as is a bit misleading here. I will comment on the etcd issue above and file a PR for the K8s watcher implementation to make use of as it is still a useful option when (re)establishing the stream. Depending on the environment, API server instances could have a health/liveness probe (or systemd equivalent) which could restart the process in case of a partition. Upon reconnect to the etcd node with the option it would immediately fail and error out accordingly.

Related discussion:
text: // The filter doesn't filter out any object. wc.internalPred = storage.Everything } <del> wc.ctx, wc.cancel = context.WithCancel(ctx) </del> <ins> // The etcd server waits until it cannot find a leader for 3 election // timeouts to cancel existing streams. 3 is currently a hard coded // constant. The election timeout defaults to 1000ms. If the cluster is // healthy, when the leader is stopped, the leadership transfer should be // smooth. (leader transfers its leadership before stopping). If leader is // hard killed, other servers will take an election timeout to realize // leader lost and start campaign. wc.ctx, wc.cancel = context.WithCancel(clientv3.WithRequireLeader(ctx)) </ins> return wc }