Change:
Hi, running 1.6.4 on etcd 3.0.17 in the following setup: 3 etcd nodes (bootstrapped according to the ) 9 kubernetes cluster nodes all of them running kube-apiserver (--apiserver-count=9) with --bind-address to local kube-apiserver instance. All instances pointing to the same etcd cluster (--etcd-servers=http://IP1:4001,http://IP2:4001,http://IP3:4001) I tried to simulate 1 etcd node crash by simply turing off one of the machines. Result: timeouts running kubectl (get nodes, get pods) commands lots of these error messages in kube-apiserver logs: "] apiserver received an error that is not an metav1.Status: http: Handler timeout" ETCD correctly reports unhealthy node, but healthy cluster Inspecting netstat shows lots of ESTABLISHED connections to the dead etcd server, port 4001 - these connections are cleared only by restarting the kube-apiserver(s). Workarounds: Switching back to etcd 2.2.1 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds Leaving etcd 3.0.17 and using --storage-backend=etcd2 ... ESTABLISHED connections are cleared within several seconds In both above cases kubectl shows expected results. The dead node is evicted (NotReady), other nodes & pods are in tact. I noticed that client contains TCP KeepAlive timeout, while client does not. Could this be related to the described scenario? <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version Client Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:44:27Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"6", GitVersion:"v1.6.4", GitCommit:"", GitTreeState:"clean", BuildDate:"2017-05-19T18:33:17Z", GoVersion:"go1.7.5", Compiler:"gc", Platform:"linux/amd64"} Environment: OS: NAME="Oracle Linux Server" VERSION="6.7" ID="ol" VERSIONID="6.7" PRETTYNAME="Oracle Linux Server 6.7" ANSICOLOR="0;31" CPENAME="cpe:/o:oracle:linux:6:7" HOMEURL="" BUGREPORTURL="" ORACLEBUGZILLAPRODUCT="Oracle Linux 6" ORACLEBUGZILLAPRODUCTVERSION=6.7 ORACLESUPPORTPRODUCT="Oracle Linux" ORACLESUPPORTPRODUCTVERSION=6.7 Kernel
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
/area etcd
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
This seems to depend on an issue with etcd, there is no option for a KeepAlive I believe. I've sent a PR, can fix this once that gets merged.
The mentioned PR has been merged. We may need to wait for an etcd release / update the dependency on etcd for us to be able to fix this in Kubernetes.
Had a look at the current CHANGELOG on etcd. The fix for this issue is in master, and will be released in etcd clientv3 3.3.0. Here is the which shows the fix has been merged into the . Kubernetes is currently on clientv3 3.2.11 it appears (according to ).

Discussion:
package factory import ( <ins> "time" </ins> "github.com/coreos/etcd/clientv3" "github.com/coreos/etcd/pkg/transport" "golang.org/x/net/context"

"k8s.io/apiserver/pkg/storage/value" ) <ins> // The short keepalive timeout and interval have been chosen to aggressively // detect a failed etcd server without introducing much overhead. var ( keepaliveTime    = 30 * time.Second keepaliveTimeout = 10 * time.Second ) </ins> func newETCD3Storage(c storagebackend.Config) (storage.Interface, DestroyFunc, error) { tlsInfo := transport.TLSInfo{ CertFile: c.CertFile,

tlsConfig = nil } cfg := clientv3.Config{ <del> Endpoints: c.ServerList, TLS:       tlsConfig, </del> <ins> DialKeepAliveTime:    keepaliveTime, DialKeepAliveTimeout: keepaliveTimeout, Endpoints:            c.ServerList, TLS:                  tlsConfig, </ins> } client, err := clientv3.New(cfg) if err != nil {