Code change:
How to reproduce the problem: Set up a new demo cluster with kubeadm 1.13.1. Create default configurationwith Initialize cluster as usual with Change the list in kube-apiserver manifest to , so that the first etcd node is unavailable ("connection refused"). The kube-apiserver is then not able to connect to etcd any more. Last message: kube-apiserver does not start. If I upgrade etcd to version 3.3.10, it reports an error Environment: Kubernetes version 1.13.1 kubeadm in Vagrant box I also experience this bug in an environment with a real etcd cluster. <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
/sig api-machinery
/remove-sig api-machinery /sig cluster-lifecycle
/sig api-machinery apologies, just had another look and it's indeed an api-machinery issue. // that can be used to communicate with a logical etcd cluster. For // example, a three-node cluster could be provided like so: // // Endpoints: []string{ // "", // "", // "", // } // // If multiple endpoints are provided, the Client will attempt to // use them all in the event that one or more of them are unusable. // // If Client.Sync is ever called, the Client may cache an alternate // set of endpoints to continue operation. we are passing the server list straight into etcd v3 client which return the error u reported. not sure if it's designed
This is an etcdv3 client issue. See
/cc
/assign So live updating a static pod manifest is typically not recommended, was this triggered via some other operation or were you editing your static manifests?
No pod manifest involved here. Just a group of etcd and a kube-apiserver. The issue appeared when we rebooted the first etcd node.
I was able to repro this issue with the repro steps provided by I also reproduced this issue with a real etcd cluster. As previously mentioned, the problem is caused by a bug in the etcd v3 client library (or perhaps the grpc library). The vault project is also running into this: The problem seems to be that the etcd library uses the first node’s address as the for TLS. This means that all attempts to connect to any server other than the first will fail with a certificate validation error (i.e. cert has in SANs, but the client is expecting ). An important thing to highlight is that when the first etcd server goes down, it also takes the Kubernetes API servers down, because they fail to connect to the remaining etcd servers. With that said, this all depends on what your etcd server certificates look like: If you follow the to stand up a 3 node etcd cluster, you get a set of certificates that include the first node’s name and IP in the SANs (because all certs are generated on the first etcd node). Thus, you should not run into this issue. If you have used another process to generate certificates for etcd, and the certs do not include the first node’s name and IP in the SANs, you will most likely run into this issue when the first etcd node goes down. To reproduce the issue with a real etcd cluster: a 3 node etcd cluster with TLS enabled. Each certificate should only contain the name/IP of the node that will be serving it. an API server that points to the etcd cluster. the first etcd node. server crashes and fails to come back up Versions: kubeadm version: v1.13.2 kubernetes api server version: v1.13.2 etcd image: API server crash log: I was unable to reproduce this issue with API server v1.12.5 (n.b. this was somewhat of a non-scientific test =tested by updating the image field of the API server static pod produced by kubeadm v1.13.2)
/assign
^ FYI.
thank you for the investigation
No problem! I started looking into the etcd client to see if I could spot the issue, but couldn't quite figure it out.
I will look into this.
I think this was the issue more on how gRPC balancer does failover with credentials. I've shared a workaround to fix this issue in upstream gRPC And waiting for their feedback. /cc
We have 3 master and 3 etcdservers, a workaround is to change the order of etcdservers. master0: master1: master2:
I just tried using a SAN cert with my etcd nodes having the names of all etcd nodes. The apiserver connects to only the last etcd node in it's list. We run 3 apiservers per cluster and I can confirm the behavior the apiservers in this particular cluster is consistent that they are picking the last, however if I shutdown the last node the connect to the other 2 without issues, once that comes back up they connect to only the last. and there are entries like this: in the etcd journal, where is the apiserver IP. Without the SAN certs the behavior matches what is described by others here. So the SAN does not make the client distribute load across etcd nodes, it just prevents a total outage if the node the api-server talking to goes down.
- what's the state here, this is still an issue.
Can we ping gRPC team again? Open to discuss over the conference call.
Will do!
Just discussed with gRPC team and We will rework on this in the next few weeks.
hi we are trying to create etcd cluster but we are facing the following error please find below error Error: client: etcd cluster is unavailable or misconfigured error : dial tcp 192.168.2.139:2379: getsockopt: connection refused please help us to get out of this thanks in advance
I'm running kubernetes 1.11.9 and etcd 3.2.24 , running on AWS provisioned by KOPS 1.12.0 and im seeing the same error message : Unable to create storage backend: config (&{etcd3 /registry [http://127.0.0.1:4001] true false 1000 0xc42039fa00 <nil5m0s 1m0s}), err (dial tcp 127.0.0.1:4001: connect: connection refused)
- any update?
I just came back from trip. Will start working on this from this week! And post updates here.
Made some progress here
If anyone needs an urgent fix, please use this -
Is Kubernetes 1.13 or 1.14 going to get updated with the fixed grpc lib? Or it's going to 1.15 only?
We are planning to backport the fix to etcd 3.3 after etcd 3.4 release. Then, kubernetes can pick up the latest etcd 3.3.
Can we please wrap up the backport (to etcd 3.3) within the next week or two? Please see timeline for 1.16 ( ) We need to have sufficient soak time for this change in k/k.
Sure. The earliest we can do is August 13, 2019. need gRPC 1.23 to include upstream bug fixes need a day or two to rewrite 3.3 client balancer Today, we do etcd 3.4 release code freeze, which means we will start running functional (failure injection) tests + kubemark to test new etcd client changes including If anything changes, we will post updates here.
Thanks a ton !
Hi, is this the same issue: I have 3 control planes. On every control plane are one etcd-server. The apiserver can only connect to one etcd-server. If I enter more then one etcd server in the kube- I have Warnings in the log: ] grpc: addrConn.createTransport failed to connect to {172.22.75.8:2379 0 <nil}. Err :connection error: desc = "transport: authentication handshake f ailed: x509: certificate is valid for 172.22.75.8, 127.0.0.1, ::1, 172.22.75.8, not 172.22.75.7". Reconnecting... The certificates are correct. I have this check with curl -v --cert apiserver-etcd- --key apiserver-etcd- --cacert ...
a quick clarification, is August 13, 2019 the earliest day the fix going to land on 3.3? thx
Yes
Update: is out. Bumping up gRPC in etcd master branch, in addition to Go runtime upgrade !topic/golang-announce/65QixT3tcmg. Once tests look good, we will start working on 3.3 backports.
has been released with all the fixes. Please try. Once tests look good in the next few days, I will release . Update: has been released.
i just kicked off a WIP on the k/k CI -
Some tests in k8s may break. We need pass with the new etcd client. Please take a look at
Is there a hotfix for v1.15?
- see
The issue is still there with Kubernetes 1.14.6 and etcd 3.3.15. Any changes to kubernetes libs or code needed to tackle this issue?
- see
hmm, if there are changes required in k8s source code, why has this issue got closed? I'll try to build a custom image and test, though. Thanks.
this will be resolved in 1.16 in the changes required (upgrading etcd client libraries) were too invasive to backport to patch releases
that leaves people on 1.13-1.15 without proper H/A? I think this issue deserves to be fixed in the three supported releases of Kubernetes. The hotfix mentioned here looks simple enough to be , but you are saying the proper fix will require much more. So this is all sorta obscure and confusing to the community, IMO. Not that I am complaining, don't get me wrong, it's just I think everyone would welcome some clarity on this issue. Maybe document it somewhere and provide some workarounds for the people who are still on v1.13.-1.15? Because right now, bring down the first etcd member and oops, api is not working, cluster is not working.
agree on documenting the issue at the very least, and recommending any possible workarounds any feedback on ^?
I will work on the documentation.
just built custom 1.14 image from the release-1.14 branch and patched that file and it's so much better now when I bring down the first etcd node down. Now I am confused, if the fix is so simple, why does it have to wait until 1.16?
The fix requires upgrading from v1.7.5 to v1.23.0, which is quite a big change :0
I am adding "Known issue" section to etcd docs here
see what said :)
Some days ago I had opened this issue , but reading this one I think it is related. Can anyone take a look at my issue and see it they are related and if so I can close it.
I believe I am running into this issue or at least something similar. I see the etcd going up and down and the api server. This cluster was created with . The logs from etcd show the following. The logs from the api server show the following. Is this the same?
There are claims here that the bug is solved, but I am seeing evidence of it not being solved in our cluster: Are we absolutely sure the etcd client fix made it onto the release? I am testing v1.6.2.
That bug is not fixed yet. The only fix was for IP address-only connections, not those using DNS names like this. We are waiting on for what will probably be Kubernetes version 1.16.3. The workaround I'm using today is to replace my etcd server certificates with ones that use a wildcard SAN for the members in the subdomain, rather than including the given machine's DNS name as a SAN. So far, it works.
It was fixed for IP addresses, but not DNS names (DNS name issue is tracked in ). Additionally, part of the fix regressed IPv6 address handling (). See These two issues have been resolved in master, and is open to pick them to 1.16 (targeting 1.16.3)
Ah... thank you and that explains it. Cheers!
Would you please explain how to change the SAN on the etcd certificates?
I generate these certificates myself using , so it's a matter of revising the arguments passed for .
Tried with Kubernetes 1.15.3 and with 1.16.2 but its not working with neither. This is not fixed even for IP addresses:
I have similar observation as with IP addresses: etcd 3.3.15 kubernetes 1.16.2 kube-apiserver log:
You should be fine with 1.16.2 and Etcd 3.3.15 now. I managed to get 3 masters running.
In my case, apiserver has been repeating warnings about connecting to external etcd cluster with tls, log snippets as follows My environment: HA kubernetes 1.15.5 cluster made by kubeadm Etcd 3.3.10 cluster with three members but i am not sure whether my issue is releated with grpc. any answer will be apprecicated
COMPLETE BACKUP AND RESTORE PROCEDURE FOR ETCD NOTE: Check that in the file "" there is the port with the address configured like this below : --listen-client-urls=https://127.0.0.1:2379,https://.......2379 1) Backup ETCDCTLAPI=3 etcdctl --endpoints=https://[127.0.0.1]:2379 -- -- -- snapshot save /tmp/snapshot-pre- NOTE: etcdctl is a command normally found on the master 2) Restore ( for restore the parameter of "--initial-cluster-token" you can call it whatever you want ) ETCDCTLAPI=3 etcdctl --endpoints=https://[127.0.0.1]:2379 -- --name=master -- -- --data-dir /var/lib/etcd-from-backup --initial-cluster=master=https://127.0.0.1:2380 --initial-cluster-token etcd-cluster-1 --initial-advertise-peer-urls=https://127.0.0.1:2380 snapshot restore /tmp/snapshot-pre- 3) Change the parameters in the following file vi --data-dir=/var/lib/etcd-from-backup ## Update --data-dir to use new target location (put in the previous restore command) --initial-cluster-token=etcd-cluster-1 ## (put in the previous restore command) volumeMounts: mountPath: /var/lib/etcd-from-backup ## changes with the path set in the previous restore command name: etcd-data hostPath: path: /var/lib/etcd-from-backup ## changes with the path set in the previous restore command type: DirectoryOrCreate name: etcd-data 4) Check the kubernetes environment after the changes See if the container process is back on docker ps -a | grep etcd see if the cluster members have been recreated ETCDCTL_API=3 etcdctl member list -- -- -- --endpoints=127.0.0.1:2379 see if pods, deployments and services have been recreated kubectl get pods,svc,deployments
"""" INSTALL KUBERNETES WITH KUBEADM """" !!! CHECK ALL INSTALLATION PREREQUISITES BEFORE INSTALLING kubernetes ---- !!! prerequisites: Check if it is already installed by running these commands on master nodes: kubectl kubeadm prerequisites: check which version of linux you have: --cat /etc/os-release prerequisites: Letting iptables see bridged traffic prerequisites: Check required ports prerequisites: install docker on all nodes if not already installed -- docker installation: sudo -i apt-get update && apt-get install -y  apt-transport-https ca-certificates curl software-properties-common gnupg2 curl -fsSL | apt-key add - add-apt-repository  "deb [arch=amd64]  $(lsbrelease -cs)  stable" apt-get update && apt-get install -y docker-ce=5:19.03.113-0ubuntu-(lsbrelease -cs) cat <<EOF { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2" } EOF mkdir -p /etc/systemd/system/docker.service.d systemctl daemon-reload systemctl restart docker per vedere se docker è attivo: systemctl status docker.service ############################end pre-requisites############################################## INSTALL KUBERNETES Return to the manual -- and install the following components: kubeadm: the command to bootstrap the cluster. kubelet: the component that runs on all of the machines in your cluster and does things like starting pods and containers. kubectl: the command line util to talk to your cluster. I report below the steps of the manual, do it on each node: sudo apt-get update && sudo apt-get install -y apt-transport-https curl curl -s | sudo apt-key add - cat <<EOF | sudo tee deb kubernetes-xenial main EOF sudo apt-get update sudo apt-get install -y kubelet kubeadm kubectl sudo apt-mark hold kubelet kubeadm kubectl systemctl daemon-reload systemctl restart kubelet Vedere la versione di kubeadm --kubeadm version -o short Vedere la versione di kubelet --kubelet --version we won't install "Configure cgroup driver" as you do when you don't have docker installed Go to the bottom of the link page mentioned above "What's next": will take you to the following link -- Give this command only on the master: do it with normal user kubeadm init Once installed, copy the output that appears and create the directories as follows on the master only: do it with normal user mkdir -p $ sudo cp -i $ sudo chown (id -g) $ generate the token on the master --"kubeadm token create --print-join-command" and then copy the output to all workers Still on the master, give the following command: kubectl get nodes ## you will see that the nodes are not active since the network for the nodes and pods has not been installed ENABLE THE NETWORK: user root: Enable the following file /proc/sys/net/bridge/bridge-nf-call-iptables to "1" for all CNI PLUGINS !!! by running this command below on all nodes: sysctl Install the network on the master with the normal user kubectl apply -f "$(kubectl version tr -d 'n')" Check again if the nodes are in the running state watch kubectl get nodes cat | grep -i image ## see the etcd version on master, then download it with the website instructions below wget -q --show-progress --https-only --timestamping "" tar -xvf etcd-v3.4.13-linux- sudo mv etcd-v3.4.13-linux-amd64/etcd /usr/local/bin/ cd /usr/local/bin/ chown root:root etcd check that everything is working ETCDCTLAPI=3 etcdctl member list --endpoints=https://127.0.0.1:2379 -- -- --
Installation of the METRICS SERVER on a MASTER without minkube copy the link of the zip file 1) wget 2) Enter on directory "assignment...." and subdirectory "metrics-server" 3) kubectl create -f deploy/1.8+/ 4) wait 5) kubectl top nodes && kubectl top pods
Deploying flannel network manually Flannel can be to any existing Kubernetes cluster though it's simplest to add flannel before any pods using the pod network have been started. For Kubernetes v1.17+ kubectl apply -f 1) ## 2) Initialize the master using the following command --kubeadm init --pod-network-cidr=192.168.0.0/16 3) Execute the following commands to configure kubectl (also returned by kubeadm init) mkdir -p $ sudo cp -i $ sudo chown (id -g) $ 4) Install the Tigera Calico operator and custom resource definitions kubectl create -f 5) Install Calico by creating the necessary custom resource kubectl create -f Note: Before creating this manifest, read its contents and make sure its settings are correct for your environment. For example, you may need to change the default IP pool CIDR to match your pod network CIDR 6) Confirm that all of the pods are running with the following command. watch kubectl get pods -n calico-system Note: The Tigera operator installs resources in the calico-system namespace. Other install methods may use the kube-system namespace instead 7) Remove the taints on the master so that you can schedule pods on it kubectl taint nodes --all node- 8) Confirm that you now have a node in your cluster with the following command. kubectl get nodes -o wide It should return something like the following. NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME <your-hostnameReady master 52m v1.12.2 10.128.0.28 <noneUbuntu 18.04.1 LTS 4.15.0-1023-gcp docker://18.6.1 About installing calicoctl calicoctl allows you to create, read, update, and delete Calico objects from the command line. Calico objects are stored in one of two datastores, either etcd or Kubernetes. The choice of datastore is determined at the time Calico is installed. Typically for Kubernetes installations the Kubernetes datastore is the default. You can run calicoctl on any host with network access to the Calico datastore as either a binary or a container. For step-by-step instructions, refer to the section that corresponds to your desired deployment. Installing calicoctl as a Kubernetes pod Use the YAML that matches your datastore type to deploy the calicoctl container to your nodes. etcd kubectl apply -f Kubernetes API datastore kubectl apply -f Note: You can also view the YAML in a new tab. You can then run commands using kubectl as shown below. kubectl exec -ti -n kube-system calicoctl -- /calicoctl get profiles -o wide An example response follows. NAME TAGS kns.default kns.default We recommend setting an alias as follows. alias calicoctl="kubectl exec -i -n kube-system calicoctl -- /calicoctl" Note: In order to use the calicoctl alias when reading manifests, redirect the file into stdin, for example: calicoctl create -f - <
we are using kops. can someone help me on this showing errors when checking systemctl status kubelet Jan 07 00:42:11 ip-172-50-2-100 kubelet[5299]: E0107 00:42:11. 5299 ] node "ip-172-50-2-100.ec2.internal" not found Jan 07 00:42:12 ip-172-50-2-100 kubelet[5299]: E0107 00:42:12. 5299 ] node "ip-172-50-2-100.ec2.internal" not found Jan 07 00:42:12 ip-172-50-2-100 kubelet[5299]: W0107 00:42:12. 5299 ] Failed to create summary reader for "": none of the resources are being tracked.

Related discussion:
text: ## Syntax and character set _Labels_ are key value pairs. Valid label keys have two segments: an optional prefix and name, separated by a slash (`/`).  The name segment is required and must be 63 characters or less, beginning and ending with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between.  The prefix is optional.  If specified, the prefix must be a DNS subdomain: a series of DNS labels separated by dots (`.`), not longer than 253 characters in total, followed by a slash (`/`). <del> If the prefix is omitted, the label key is presumed to be private to the user. System components which use labels must specify a prefix.  The `kubernetes.io/` prefix is reserved for kubernetes core components. </del> <ins> If the prefix is omitted, the label key is presumed to be private to the user. Automated system components (e.g. ```kube-scheduler```, ```kube-controller-manager```, ```kube-apiserver```, ```kubectl```, or other third-party automation) which add labels to end-user objects must specify a prefix.  The `kubernetes.io/` prefix is reserved for kubernetes core components. </ins> Valid label values must be 63 characters or less and must be empty or begin and end with an alphanumeric character (`[a-z0-9A-Z]`) with dashes (`-`), underscores (`_`), dots (`.`), and alphanumerics between.