Code change:
<!-- ## Securing etcd clusters <del> Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters. To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication. </del> <ins> Access to etcd is equivalent to root permission in the cluster so ideally only the API server should have access to it. Considering the sensitivity of the data, it is recommended to grant permission to only those nodes that require access to etcd clusters. To secure etcd, either set up firewall rules or use the security features provided by etcd. etcd security features depend on x509 Public Key Infrastructure (PKI). To begin, establish secure communication channels by generating a key and certificate pair. For example, use key pairs `peer.key` and `peer.cert` for securing communication between etcd members, and `client.key` and `client.cert` for securing communication between etcd and its clients. See the [example scripts](https://github.com/coreos/etcd/tree/master/hack/tls-setup) provided by the etcd project to generate key pairs and CA files for client authentication. </ins> --> ## 安全的 etcd 集群 <del> 对 etcd 的访问相当于集群中的 root 权限，因此理想情况下只有 API 服务器才能访问它。考虑到数据的敏感性，建议只向需要访问 etcd 集群的节点授予权限。 </del> <ins> 对 etcd 的访问相当于集群中的 root 权限，因此理想情况下只有 API 服务器才能访问它。 考虑到数据的敏感性，建议只向需要访问 etcd 集群的节点授予权限。 </ins> <del> 想要确保 etcd 的安全，可以设置防火墙规则或使用 etcd 提供的安全特性，这些安全特性依赖于 x509 公钥基础设施（PKI）。首先，通过生成密钥和证书对来建立安全的通信通道。 例如，使用密钥对 `peer.key` 和 `peer.cert` 来保护 etcd 成员之间的通信，而 `client.cert` 和 `client.cert` 用于保护 etcd 与其客户端之间的通信。请参阅 etcd 项目提供的[示例脚本](https://github.com/coreos/etcd/tree/master/hack/tls-setup)，以生成用于客户端身份验证的密钥对和 CA 文件。 </del> <ins> 想要确保 etcd 的安全，可以设置防火墙规则或使用 etcd 提供的安全特性，这些安全特性依赖于 x509 公钥基础设施（PKI）。 首先，通过生成密钥和证书对来建立安全的通信通道。 例如，使用密钥对 `peer.key` 和 `peer.cert` 来保护 etcd 成员之间的通信， 而 `client.key` 和 `client.cert` 用于保护 etcd 与其客户端之间的通信。 请参阅 etcd 项目提供的[示例脚本](https://github.com/coreos/etcd/tree/master/hack/tls-setup)， 以生成用于客户端身份验证的密钥对和 CA 文件。 </ins> <!-- ### Securing communication <del> To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use https as URL schema. </del> <ins> To configure etcd with secure peer communication, specify flags `--peer-key-file=peer.key` and `--peer-cert-file=peer.cert`, and use HTTPS as the URL schema. Similarly, to configure etcd with secure client communication, specify flags `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use HTTPS as the URL schema. Here is an example on a client command that uses secure communication: </ins> <del> Similarly, to configure etcd with secure client communication, specify flags `--key-file=k8sclient.key` and `--cert-file=k8sclient.cert`, and use https as URL schema. </del> <ins> ``` ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379  --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key  --cacert=/etc/kubernetes/pki/etcd/ca.crt  member list ``` </ins> --> ### 安全通信 <del> 若要使用安全对等通信对 etcd 进行配置，请指定参数 `--peer-key-file=peer.key` 和 `--peer-cert-file=peer.cert`，并使用 https 作为 URL 模式。 </del> <ins> 若要使用安全对等通信对 etcd 进行配置，请指定参数 `--peer-key-file=peer.key` 和 `--peer-cert-file=peer.cert`，并使用 HTTPS 作为 URL 模式。 </ins> <del> 类似地，要使用安全客户端通信对 etcd 进行配置，请指定参数 `--key-file=k8sclient.key` 和 `--cert-file=k8sclient.cert`，并使用 https 作为 URL 模式。 </del> <ins> 类似地，要使用安全客户端通信对 etcd 进行配置，请指定参数 `--key-file=k8sclient.key` 和 `--cert-file=k8sclient.cert`，并使用 HTTPS 作为 URL 模式。 使用安全通信的客户端命令的示例： ``` ETCDCTL_API=3 etcdctl --endpoints 10.2.0.9:2379  --cert=/etc/kubernetes/pki/etcd/server.crt  --key=/etc/kubernetes/pki/etcd/server.key  --cacert=/etc/kubernetes/pki/etcd/ca.crt  member list ``` </ins> <!-- ### Limiting access of etcd clusters <del> After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API server. Use TLS authentication to do so. </del> <ins> After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API servers. Use TLS authentication to do so. </ins> <del> For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trusted-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`. </del> <ins> For example, consider key pairs `k8sclient.key` and `k8sclient.cert` that are trusted by the CA `etcd.ca`. When etcd is configured with `--client-cert-auth` along with TLS, it verifies the certificates from clients by using system CAs or the CA passed in by `--trusted-ca-file` flag. Specifying flags `--client-cert-auth=true` and `--trusted-ca-file=etcd.ca` will restrict the access to clients with the certificate `k8sclient.cert`. </ins> <del> Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API server the access, configure it with the flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and `--etcd-cafile=ca.cert`. </del> <ins> Once etcd is configured correctly, only clients with valid certificates can access it. To give Kubernetes API servers the access, configure them with the flags `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` and `--etcd-cafile=ca.cert`. </ins> {{< note >}} <del> etcd authentication is not currently supported by Kubernetes. For more information, see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398). </del> <ins> etcd authentication is not currently supported by Kubernetes. For more information, see the related issue [Support Basic Auth for Etcd v2](https://github.com/kubernetes/kubernetes/issues/23398). </ins> {{< /note >}} --> ### 限制 etcd 集群的访问 配置安全通信后，将 etcd 集群的访问限制在 Kubernetes API 服务器上。使用 TLS 身份验证来完成此任务。 <del> 例如，考虑由 CA `etcd.ca` 信任的密钥对 `k8sclient.key` 和 `k8sclient.cert`。当 etcd 配置为 `--client-cert-auth` 和 TLS 时，它使用系统 CA 或由 `--trusted-ca-file` 参数传入的 CA 验证来自客户端的证书。 </del> <ins> 例如，考虑由 CA `etcd.ca` 信任的密钥对 `k8sclient.key` 和 `k8sclient.cert`。 当 etcd 配置为 `--client-cert-auth` 和 TLS 时，它使用系统 CA 或由 `--trusted-ca-file` 参数传入的 CA 验证来自客户端的证书。 </ins> 指定参数 `--client-cert-auth=true` 和 `--trusted-ca-file=etcd.ca` 将限制对具有证书 `k8sclient.cert` 的客户端的访问。 <del> 一旦正确配置了 etcd，只有具有有效证书的客户端才能访问它。要让 Kubernetes API 服务器访问，可以使用参数 `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` 和 `--etcd-cafile=ca.cert` 配置它。 </del> <ins> 一旦正确配置了 etcd，只有具有有效证书的客户端才能访问它。要让 Kubernetes API 服务器访问， 可以使用参数 `--etcd-certfile=k8sclient.cert`,`--etcd-keyfile=k8sclient.key` 和 `--etcd-cafile=ca.cert` 配置。 </ins> {{< note >}} <del> Kubernetes 目前不支持 etcd 身份验证。想要了解更多信息，请参阅相关的问题[支持 etcd v2 的基本认证](https://github.com/kubernetes/kubernetes/issues/23398)。 </del> <ins> Kubernetes 目前不支持 etcd 身份验证。 想要了解更多信息，请参阅相关的问题 [支持 etcd v2 的基本认证](https://github.com/kubernetes/kubernetes/issues/23398)。 </ins> {{< /note >}} <!-- ## Replacing a failed etcd member <del> etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member. </del> <ins> etcd cluster achieves high availability by tolerating minor member failures. However, to improve the overall health of the cluster, replace failed members immediately. When multiple members fail, replace them one by one. Replacing a failed member involves two steps: removing the failed member and adding a new member. </ins> <del> Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, member1=http://10.0.0.1, member2=http://10.0.0.2, and member3=http://10.0.0.3. When member1 fails, replace it with member4=http://10.0.0.4. </del> <ins> Though etcd keeps unique member IDs internally, it is recommended to use a unique name for each member to avoid human errors. For example, consider a three-member etcd cluster. Let the URLs be, `member1=http://10.0.0.1`, `member2=http://10.0.0.2`, and `member3=http://10.0.0.3`. When `member1` fails, replace it with `member4=http://10.0.0.4`. </ins> <del> 1. Get the member ID of the failed member1: </del> <ins> 1. Get the member ID of the failed `member1`: </ins> <del> `etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list` </del> <ins> ```shell etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 </del> <ins> ```console 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 ``` </ins> 2. Remove the failed member: <del> `etcdctl member remove 8211f1d0f64f3269` </del> <ins> ```shell etcdctl member remove 8211f1d0f64f3269 ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> Removed member 8211f1d0f64f3269 from cluster </del> <ins> ```console Removed member 8211f1d0f64f3269 from cluster ``` </ins> 3. Add the new member: <del> `./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380` </del> <ins> ```shell etcdctl member add member4 --peer-urls=http://10.0.0.4:2380 ``` </ins> <del> The following message is displayed: </del> <ins> The following message is displayed: </ins> <del> Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 </del> <ins> ```console Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 ``` </ins> 4. Start the newly added member on a machine with the IP `10.0.0.4`: <del> export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] </del> <ins> ```shell export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] ``` </ins> 5. Do either of the following: <del> 1. Update its `--etcd-servers` flag to make Kubernetes aware of the configuration changes, then restart the Kubernetes API server. 2. Update the load balancer configuration if a load balancer is used in the deployment. </del> <ins> 1. Update the `--etcd-servers` flag for the Kubernetes API servers to make Kubernetes aware of the configuration changes, then restart the Kubernetes API servers. 2. Update the load balancer configuration if a load balancer is used in the deployment. </ins> <del> For more information on cluster reconfiguration, see [etcd Reconfiguration Documentation](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member). </del> <ins> For more information on cluster reconfiguration, see [etcd reconfiguration documentation](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member). </ins> --> ## 替换失败的 etcd 成员 <del> etcd 集群通过容忍少数成员故障实现高可用性。但是，要改善集群的整体健康状况，请立即替换失败的成员。当多个成员失败时，逐个替换它们。替换失败成员需要两个步骤：删除失败成员和添加新成员。 </del> <ins> etcd 集群通过容忍少数成员故障实现高可用性。 但是，要改善集群的整体健康状况，请立即替换失败的成员。当多个成员失败时，逐个替换它们。 替换失败成员需要两个步骤：删除失败成员和添加新成员。 </ins> <del> 虽然 etcd 在内部保留唯一的成员 ID，但建议为每个成员使用唯一的名称，以避免人为错误。例如，考虑一个三成员的 etcd 集群。让 URL 为：member1=http://10.0.0.1， member2=http://10.0.0.2 和 member3=http://10.0.0.3。当 member1 失败时，将其替换为 member4=http://10.0.0.4。 </del> <ins> 虽然 etcd 在内部保留唯一的成员 ID，但建议为每个成员使用唯一的名称，以避免人为错误。 例如，考虑一个三成员的 etcd 集群。让 URL 为：`member1=http://10.0.0.1`， `member2=http://10.0.0.2` 和 `member3=http://10.0.0.3`。当 `member1` 失败时，将其替换为 `member4=http://10.0.0.4`。 </ins> <del> 1. 获取失败的 member1 的成员 ID： </del> <ins> 1. 获取失败的 `member1` 的成员 ID： </ins> <del> `etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list` </del> <ins> ```shell etcdctl --endpoints=http://10.0.0.2,http://10.0.0.3 member list ``` </ins> <del> 显示以下信息： </del> <ins> 显示以下信息： </ins> <del> 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 </del> <ins> ```console 8211f1d0f64f3269, started, member1, http://10.0.0.1:2380, http://10.0.0.1:2379 91bc3c398fb3c146, started, member2, http://10.0.0.2:2380, http://10.0.0.2:2379 fd422379fda50e48, started, member3, http://10.0.0.3:2380, http://10.0.0.3:2379 ``` </ins> 2. 移除失败的成员 <del> `etcdctl member remove 8211f1d0f64f3269` </del> <ins> ```shell etcdctl member remove 8211f1d0f64f3269 ``` </ins> <del> 显示以下信息： </del> <ins> 显示以下信息： </ins> <del> Removed member 8211f1d0f64f3269 from cluster </del> <ins> ```console Removed member 8211f1d0f64f3269 from cluster ``` </ins> 3. 增加新成员： <del> `./etcdctl member add member4 --peer-urls=http://10.0.0.4:2380` </del> <ins> ```shell etcdctl member add member4 --peer-urls=http://10.0.0.4:2380 ``` </ins> <del> 显示以下信息： </del> <ins> 显示以下信息： </ins> <del> Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 </del> <ins> ```console Member 2be1eb8f84b7f63e added to cluster ef37ad9dc622a7c4 ``` </ins> 4. 在 IP 为 `10.0.0.4` 的机器上启动新增加的成员： <del> export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] </del> <ins> ```shell export ETCD_NAME="member4" export ETCD_INITIAL_CLUSTER="member2=http://10.0.0.2:2380,member3=http://10.0.0.3:2380,member4=http://10.0.0.4:2380" export ETCD_INITIAL_CLUSTER_STATE=existing etcd [flags] ``` </ins> 5. 做以下事情之一： <del> 1. 更新其 `--etcd-servers` 参数，使 Kubernetes 知道配置进行了更改，然后重新启动 Kubernetes API 服务器。 </del> <ins> 1. 更新 Kubernetes API 服务器的 `--etcd-servers` 参数，使 Kubernetes 知道配置进行了更改，然后重新启动 Kubernetes API 服务器。 </ins> 2. 如果在 deployment 中使用了负载均衡，更新负载均衡配置。 <del> 有关集群重新配置的详细信息，请参阅 [etcd 重构文档](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/runtime-configuration.md#remove-a-member)。 </del> <ins> 有关集群重新配置的详细信息，请参阅 [etcd 重构文档](https://etcd.io/docs/current/op-guide/runtime-configuration/#remove-a-member)。 </ins> <!-- ## Backing up an etcd cluster <del> All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all master nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files. </del> <ins> All Kubernetes objects are stored on etcd. Periodically backing up the etcd cluster data is important to recover Kubernetes clusters under disaster scenarios, such as losing all control plane nodes. The snapshot file contains all the Kubernetes states and critical information. In order to keep the sensitive Kubernetes data safe, encrypt the snapshot files. </ins> <del> Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot. </del> <ins> Backing up an etcd cluster can be accomplished in two ways: etcd built-in snapshot and volume snapshot. </ins> --> ## 备份 etcd 集群 <del> 所有 Kubernetes 对象都存储在 etcd 上。定期备份 etcd 集群数据对于在灾难场景（例如丢失所有主节点）下恢复 Kubernetes 集群非常重要。快照文件包含所有 Kubernetes 状态和关键信息。为了保证敏感的 Kubernetes 数据的安全，可以对快照文件进行加密。 </del> <ins> 所有 Kubernetes 对象都存储在 etcd 上。定期备份 etcd 集群数据对于在灾难场景（例如丢失所有控制平面节点）下恢复 Kubernetes 集群非常重要。 快照文件包含所有 Kubernetes 状态和关键信息。为了保证敏感的 Kubernetes 数据的安全，可以对快照文件进行加密。 </ins> 备份 etcd 集群可以通过两种方式完成：etcd 内置快照和卷快照。 <!-- ### Built-in snapshot <del> etcd supports built-in snapshot, so backing up an etcd cluster is easy. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://github.com/coreos/etcd/blob/master/Documentation/op-guide/configuration.md#--data-dir) that is not currently used by an etcd process. Taking the snapshot will normally not affect the performance of the member. </del> <ins> etcd supports built-in snapshot. A snapshot may either be taken from a live member with the `etcdctl snapshot save` command or by copying the `member/snap/db` file from an etcd [data directory](https://etcd.io/docs/current/op-guide/configuration/#--data-dir) that is not currently used by an etcd process. Taking the snapshot will not affect the performance of the member. </ins> <del> Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`: </del> <ins> Below is an example for taking a snapshot of the keyspace served by `$ENDPOINT` to the file `snapshotdb`: </ins> <del> ```sh </del> <ins> ```shell </ins> ETCDCTL_API=3 etcdctl --endpoints $ENDPOINT snapshot save snapshotdb <del> # exit 0 </del> <ins> ``` </ins> <del> # verify the snapshot </del> <ins> Verify the snapshot: ```shell </ins> ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshotdb <ins> ``` ```console </ins> +----------+----------+------------+------------+ |   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE | +----------+----------+------------+------------+

Related discussion:
text: " etcd "client.cert " etcd etcd " " etcd client.key etcd "
~ GitHub is not the right place for support requests.~ ~If you're looking for help, oyu could ask on .~ ~You can also post your question on the or the forum. There are other support venues with contributions from the wider community of Kubernetes users.~
/triage accepted /kind bug
/remove-kind support
Thanks
/language zh
/priority backlog