Code change:
## Securing etcd clusters <del> Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters. To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication. </del> <ins> Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters. To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication. </ins> ### Securing communication <del> To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use https as URL schema. </del> <ins> To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as the URL schema. </ins> <del> Similarly, to configure etcd with secure client communication, specify flags `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use https as URL schema. </del> <ins> Similarly, to configure etcd with secure client communication, specify flags `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as the URL schema. </ins> ### Limiting access of etcd clusters <del> After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API server. Use TLS authentication to do so. </del> <ins> After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API servers. Use TLS authentication to do so. </ins> <del> For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trusted-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`. </del> <ins> For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trusted-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`. </ins> <del> Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API server the access, configure it with the flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and `--etcd-cafile=ca.cert`. </del> <ins> Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API servers the access, configure them with the flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and `--etcd-cafile=ca.cert`. </ins> {{< note >}} <del> etcd authentication is not currently supported by Kubernetes. For more information, see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398). </del> <ins> etcd authentication is not currently supported by Kubernetes. For more information, see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398). </ins> {{< /note >}} ## Replacing a failed etcd member <del> etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member. </del> <ins> etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member. </ins> <del> Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1, member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails, replace it with member4=http://10.0.0.4. </del> <ins> Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, `member1=http://10.0.0.1`, `member2=http://10.0.0.2`, and `member3=http://10.0.0.3`. When `member1` fails, replace it with `member4=http://10.0.0.4`. </ins> <del> 1. Get the member ID of the failed member1: </del> <ins> 1. Get the member ID of the failed `member1`: </ins> <del> `etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list` </del> <ins> ```shell etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 </del> <ins> ```console 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 ``` </ins> 2. Remove the failed member: <del> `etcdctl member remove 8211f1d0f64f3269` </del> <ins> ```shell etcdctl member remove 8211f1d0f64f3269 ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> Removed member 8211f1d0f64f3269 from cluster </del> <ins> ```console Removed member 8211f1d0f64f3269 from cluster ``` </ins> 3. Add the new member: <del> `./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380` </del> <ins> ```shell etcdctl member add member4 --peer-urls=http://10.0.0.4:2380 ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 </del> <ins> ```console Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 ``` </ins> 4. Start the newly added member on a machine with the IP `10.0.0.4`: <del> export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] </del> <ins> ```shell export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] ``` </ins> 5. Do either of the following: <del> 1. Update its `--etcd-servers` flag to make Kubernetes aware of the configuration changes, then restart the Kubernetes API server. 2. Update the load balancer configuration if a load balancer is used in the deployment. </del> <ins> 1. Update the `--etcd-servers` flag for the Kubernetes API servers to make Kubernetes aware of the configuration changes, then restart the Kubernetes API servers. 2. Update the load balancer configuration if a load balancer is used in the deployment. </ins> <del> For more information on cluster reconfiguration, see [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member). </del> <ins> For more information on cluster reconfiguration, see [etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member). </ins> ## Backing up an etcd cluster <del> All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all master nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files. </del> <ins> All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all control plane nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files. </ins> <del> Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot. </del> <ins> Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot. </ins> ### Built-in snapshot <del> etcd supports built-in snapshot. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir) that is not currently used by an etcd process. Taking the snapshot will normally not affect the performance of the member. </del> <ins> etcd supports built-in snapshot. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://etcd.io/docs/current/op-guide/configuration/#--data-dir) that is not currently used by an etcd process. Taking the snapshot will not affect the performance of the member. </ins> <del> Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`: </del> <ins> Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`: </ins> <del> ```sh </del> <ins> ```shell </ins> ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdb <del> # exit 0 </del> <ins> ``` Verify the snapshot: </ins> <del> # verify the snapshot </del> <ins> ```shell </ins> ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb <ins> ``` ```console </ins> +----------+----------+------------+------------+ |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+

Related discussion:
text: Links to github on this page seems to be broken
/triage accepted /assign
Also see issue