Repository: kubernetes/kops (go)

Issue:
single to multi-master docs suggest you can add etcdCluster memebers but Kops 1.16.0-alpha.2 says no
**1. What `kops` version are you running? The command `kops version`, will display
 this information.**
Version 1.16.0-alpha.2 (git-dfaf0b633)

**2. What Kubernetes version are you running? `kubectl version` will print the
 version if a cluster is running or provide the Kubernetes version specified as
 a `kops` flag.**
```
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T23:42:50Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
```

**3. What cloud provider are you using?**
AWS

**4. What commands did you run?  What is the simplest way to reproduce this issue?**
changed cluster config from:
```
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-ap-southeast-2c
      name: c
    name: main
  - etcdMembers:
    - instanceGroup: master-ap-southeast-2c
      name: c
    name: events
```
to 
```
  etcdClusters:
  - etcdMembers:
    - instanceGroup: master-ap-southeast-2a
      name: a
    - instanceGroup: master-ap-southeast-2b
      name: b
    - instanceGroup: master-ap-southeast-2c
      name: c
    name: main
  - etcdMembers:
    - instanceGroup: master-ap-southeast-2a
      name: a
    - instanceGroup: master-ap-southeast-2b
      name: b
    - instanceGroup: master-ap-southeast-2c
      name: c
    name: events
```

and then ran:

`kops replace -f kops/all.yaml -v 9`

**5. What happened after the commands executed?**
```
F1211 13:29:14.215685   50042 helpers.go:114] error: error replacing cluster: [spec.etcdClusters[main].Members[a]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[main].Members[b]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[events].Members[a]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[events].Members[b]: Forbidden: EtcdCluster members cannot be added]
```

**6. What did you expect to happen?**
According to the [single to multi-master docs ](https://github.com/kubernetes/kops/blob/master/docs/single-to-multi-master.md), this should work

**7. Please provide your cluster manifest. Execute
  `kops get --name my.example.com -o yaml` to display your cluster manifest.
  You may want to remove your cluster name and other sensitive information.**

**8. Please run the commands with most verbose logging by adding the `-v 10` flag.
  Paste the logs into this report, or in a gist and provide the gist link here.**
```
+ kops-darwin-amd64 replace -f kops/all.yaml --force -v 10
I1211 13:47:51.384480   50298 factory.go:68] state store s3://clusters.dev.aws.eliiza.com.au
I1211 13:47:51.414782   50298 aws_cloud.go:1229] Querying EC2 for all valid zones in region "ap-southeast-2"
I1211 13:47:51.418868   50298 request_logger.go:45] AWS request: ec2/DescribeAvailabilityZones
I1211 13:47:51.735982   50298 status.go:57] Querying AWS for etcd volumes
I1211 13:47:51.736336   50298 status.go:68] Listing EC2 Volumes
I1211 13:47:51.737796   50298 request_logger.go:45] AWS request: ec2/DescribeVolumes
I1211 13:47:52.019957   50298 status.go:40] Cluster status (from cloud): {"etcdClusters":[{"name":"events","etcdMembers":[{"name":"c","volumeId":"vol-081ebb8a13282b8b9"}]},{"name":"main","etcdMembers":[{"name":"c","volumeId":"vol-04b8919810e8a37e7"}]}]}
I1211 13:47:52.180694   50298 s3context.go:210] found bucket in region "us-east-1"
I1211 13:47:52.181117   50298 s3fs.go:220] Reading file "s3://clusters.dev.aws.eliiza.com.au/sydney.k8s.aws.eliiza.com.au/config"
I1211 13:47:53.199843   50298 s3fs.go:220] Reading file "s3://clusters.dev.aws.eliiza.com.au/sydney.k8s.aws.eliiza.com.au/config"
F1211 13:47:53.458748   50298 helpers.go:114] error: error replacing cluster: [spec.etcdClusters[events].Members[a]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[events].Members[b]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[main].Members[b]: Forbidden: EtcdCluster members cannot be added, spec.etcdClusters[main].Members[a]: Forbidden: EtcdCluster members cannot be added]
```

**9. Anything else do we need to know?**


Pull request:
Allow single to multi-master cluster
Removes some necessary restrictions and updating the docs.
Fixes #8091

Patch:
diff --git a/channels/alpha b/channels/alpha
index bdb88be302..e30c70ad52 100644
--- a/channels/alpha
+++ b/channels/alpha
@@ -56,10 +56,10 @@ spec:
       kubenet: {}
   kubernetesVersions:
   - range: ">=1.17.0"
-    recommendedVersion: 1.17.5
+    recommendedVersion: 1.17.4
     requiredVersion: 1.17.0
   - range: ">=1.16.0"
-    recommendedVersion: 1.16.9
+    recommendedVersion: 1.16.8
     requiredVersion: 1.16.0
   - range: ">=1.15.0"
     recommendedVersion: 1.15.11
@@ -83,11 +83,11 @@ spec:
   - range: ">=1.17.0-alpha.1"
     #recommendedVersion: "1.17.0"
     #requiredVersion: 1.17.0
-    kubernetesVersion: 1.17.5
+    kubernetesVersion: 1.17.4
   - range: ">=1.16.0-alpha.1"
     #recommendedVersion: "1.16.0"
     #requiredVersion: 1.16.0
-    kubernetesVersion: 1.16.9
+    kubernetesVersion: 1.16.8
   - range: ">=1.15.0-alpha.1"
     #recommendedVersion: "1.15.0"
     #requiredVersion: 1.15.0
diff --git a/channels/stable b/channels/stable
index d7ef7bacaf..9f96f03cdb 100644
--- a/channels/stable
+++ b/channels/stable
@@ -56,10 +56,10 @@ spec:
       kubenet: {}
   kubernetesVersions:
   - range: ">=1.17.0"
-    recommendedVersion: 1.17.4
+    recommendedVersion: 1.17.3
     requiredVersion: 1.17.0
   - range: ">=1.16.0"
-    recommendedVersion: 1.16.8
+    recommendedVersion: 1.16.7
     requiredVersion: 1.16.0
   - range: ">=1.15.0"
     recommendedVersion: 1.15.10
@@ -83,11 +83,11 @@ spec:
   - range: ">=1.17.0-alpha.1"
     #recommendedVersion: "1.17.0"
     #requiredVersion: 1.17.0
-    kubernetesVersion: 1.17.4
+    kubernetesVersion: 1.17.3
   - range: ">=1.16.0-alpha.1"
     #recommendedVersion: "1.16.0"
     #requiredVersion: 1.16.0
-    kubernetesVersion: 1.16.8
+    kubernetesVersion: 1.16.7
   - range: ">=1.15.0-alpha.1"
     #recommendedVersion: "1.15.0"
     #requiredVersion: 1.15.0
diff --git a/docs/single-to-multi-master.md b/docs/single-to-multi-master.md
index 9330bfb850..a28bb41735 100644
--- a/docs/single-to-multi-master.md
+++ b/docs/single-to-multi-master.md
@@ -1,93 +1,21 @@
 # Migrating from single to multi-master
 
-This document describes how to go from a single-master cluster (created by kops)
-to a multi-master cluster. If you are using etcd-manager you just need to perform some steps of the migration.  
+Switching from a single-master to a multi-maser Kubernetes cluster is an entirely graceful procedure when using etcd-manager.
+If you are still using legacy etcd, you need to migrate to etcd-manager first.
 
-# etcd-manager
+## Create instance groups
 
-If you are using etcd-manager, just perform the steps in this section. Etcd-manager is default for kops 1.12. Etcd-manager makes the upgrade to multi-master much smoother. 
+### Create new subnets
 
-The list references steps of the next section. To upgrade from a single master to a cluster with three masters:
+Start out by deciding which availability zones you want to deploy the masters to. You can only have one master per availability zone.
 
-- Skip Step 1 since Etcd-manager is doing automatic backups to S3
-- Create Instance Groups (Section 2 below)
-  - create the subnets
-  - create the instance groups (no need to disable the third master, leave minSize and maxSize at 1)
-  - add the masters to your etcd cluster definition (both in section named main and events)
-- Skip Step 3 and 4
-- Now you are ready to update the AWS configuration:
-  - `kops update cluster your-cluster-name`
-- AWS will launch two new masters, they will be discovered and then configured by etcd-manager
-- check with `kubectl get nodes` to see everything is ready
-- Cleanup (Step 5) to do a rolling restart of all masters (just in case)
-
-# Etcd without etcd-manager
-
-## 0 - Warnings
-
-This is a risky procedure that **can lead to data-loss** in the etcd cluster.
-Please follow all the backup steps before attempting it. Please read the
-[etcd admin guide](https://github.com/coreos/etcd/blob/v2.2.1/Documentation/admin_guide.md)
-before attempting it.
-
-We can migrate from a single-master cluster to a multi-master cluster, but this is a complicated operation. It is easier to create a multi-master cluster using Kops (described [here](operations/high_availability.md)). If possible, try to plan this at time of cluster creation.
-
-During this procedure, you will experience **downtime** on the API server, but
-not on the end user services. During this downtime, existing pods will continue
-to work, but you will not be able to create new pods and any existing pod that
-dies will not be restarted.
-
-## 1 - Backups
-
-### a - Backup main etcd cluster
-
-```bash
-$ kubectl --namespace=kube-system get pods | grep etcd
-etcd-server-events-ip-172-20-36-161.ec2.internal        1/1       Running   4          2h
-etcd-server-ip-172-20-36-161.ec2.internal               1/1       Running   4          2h
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -it -- sh
-/ # etcdctl backup --data-dir /var/etcd/data --backup-dir /var/etcd/backup
-/ # mv /var/etcd/backup/ /var/etcd/data/
-/ # exit
-$ kubectl --namespace=kube-system get pod etcd-server-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
-"/mnt/master-vol-0ea119c15602cbb57/var/etcd/data"
-$ ssh admin@<master-node>
-admin@ip-172-20-36-161:~$ sudo -i
-root@ip-172-20-36-161:~# mv /mnt/master-vol-0ea119c15602cbb57/var/etcd/data/backup /home/admin/
-root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup/
-root@ip-172-20-36-161:~# exit
-admin@ip-172-20-36-161:~$ exit
-$ scp -r admin@<master-node>:backup/ .
-```
-
-### b - Backup event etcd cluster
+Then you need to add new subnets for your availability zones. Which subnets you need to add depend on which topology you have chosen. Simplest is to copy the sections you already have. Make sure that you add additional subnets per type. E.g if you have a `private` and a `utility` subnet, you need to copy both.
 
 ```bash
-$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -it -- sh
-/ # etcdctl backup --data-dir /var/etcd/data-events --backup-dir /var/etcd/backup
-/ # mv /var/etcd/backup/ /var/etcd/data-events/
-/ # exit
-$ kubectl --namespace=kube-system get pod etcd-server-events-ip-172-20-36-161.ec2.internal -o json | jq '.spec.volumes[] | select(.name | contains("varetcdata")) | .hostPath.path'
-"/mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events"
-$ ssh admin@<master-node>
-admin@ip-172-20-36-161:~$ sudo -i
-root@ip-172-20-36-161:~# mv /mnt/master-vol-0bb5ad222911c6777/var/etcd/data-events/backup/ /home/admin/backup-events
-root@ip-172-20-36-161:~# chown -R admin: /home/admin/backup-events/
-root@ip-172-20-36-161:~# exit
-admin@ip-172-20-36-161:~$ exit
-$ scp -r admin@<master-node>:backup-events/ .
+kops get cluster -o yaml > mycluster.yaml
 ```
 
-## 2 - Create instance groups
-
-### a - Create new subnets
-
-Add new subnets for your availability zones. Also create subnets of type Utility if they are defined in your configuration. 
-```bash
-kops edit cluster
-```
-
-Change the subnet section. An example might be: (**Adapt to your configuration!**)
+Change the subnet section to look something like this:
 ```yaml
   - cidr: 172.20.32.0/19
     name: eu-west-1a
@@ -115,56 +43,35 @@ Change the subnet section. An example might be: (**Adapt to your configuration!*
     zone: eu-west-1c
 ```
 
-### b - Create new master instance group
+### Create new master instance groups
 
-Create 1 kops instance group for the first one of your new masters, in
-a different AZ from the existing one. 
+The next step is creating two new instance groups for the new masters. 
 
 ```bash
-$ kops create instancegroup master-<availability-zone2> --subnet <availability-zone2> --role Master
-```
-Example:
-
-```bash
-$ kops create ig master-eu-west-1b --subnet eu-west-1b --role Master
-```
-
- * ``maxSize`` and ``minSize`` should be 1,
- * only one zone should be listed.
- * adjust the machineType
- * adjust the image to the OS you are using
-
-### c - Create third master instance group
-
-Instance group for the third master, in a different AZ from the existing one, is
-also required. However, real EC2 instance is not required until the second master launches.
-
-```bash
-$ kops create instancegroup master-<availability-zone3> --subnet <availability-zone3> --role Master
+kops create instancegroup master-<subnet name> --subnet <subnet name> --role Master
 ```
 
 Example:
 
 ```bash
-$ kops create ig master-eu-west-1c --subnet eu-west-1c --role Master
+kops create ig master-us-west-1d --subnet us-west-1d --role Master
 ```
 
- * ``maxSize`` and ``minSize`` should be **0**,
- * if you are using etcd-manager, you just can leave the `maxSize` and `minSize` at **1**.
- * only one zone should be listed.
- * adjust the machineType
- * adjust the image to the OS you are using
+This command will bring up an editor with the default values. Ensure that:
 
-### d - Reference the new masters in your cluster configuration
+ * `maxSize` and `minSize` is 1
+ * only one zone is listed
+ * you have the correct image and machine type
 
-*kops will refuse to have only 2 members in the etcd clusters, so we have to
-reference a third one, even if we have not created it yet.*
+### Reference the new masters in your cluster configuration
+
+Bring up `mycluster.yaml` again to add etcd members to each of new masters.
 
 ```bash
-$ kops edit cluster example.com
+$EDITOR mycluster.yaml
 ```
 
- * In ``.spec.etcdClusters`` add 2 new members in each cluster, one for each new
+ * In `.spec.etcdClusters` add 2 new members in each cluster, one for each new
  availability zone.
 
 ```yaml
@@ -196,192 +103,26 @@ etcdClusters:
     name: events
 ```
 
-## 3 - Add a new master
-
-### a - Add a new member to the etcd clusters
-
-**The clusters will stop to work until the new member is started**.
-
-```bash
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone2-name> http://etcd-<availability-zone2-name>.internal.example.com:2380 \
-	&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone2-name> http://etcd-events-<availability-zone2-name>.internal.example.com:2381
-```
-
-Example:
-
-```bash
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-b http://etcd-b.internal.example.com:2380 \
-	&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-b http://etcd-events-b.internal.example.com:2381
-```
-
-### b - Launch the new master
-
-```bash
-$ kops update cluster example.com --yes
-# wait for the new master to boot and initialize
-$ ssh admin@<new-master>
-admin@ip-172-20-116-230:~$ sudo -i
-root@ip-172-20-116-230:~# systemctl stop kubelet
-root@ip-172-20-116-230:~# systemctl stop protokube
-```
-
-Reinitialize the etcd instances:
-* In both ``/etc/kubernetes/manifests/etcd-events.manifest`` and
-``/etc/kubernetes/manifests/etcd.manifest``, edit the
-``ETCD_INITIAL_CLUSTER_STATE`` variable to ``existing``.
-* In the same files, remove the third non-existing member from
-``ETCD_INITIAL_CLUSTER``.
-* Delete the containers and the data directories. Mount paths can be determined with the help of ``docker inspect <container-id> | grep /mnt/master-vol``:
-
-```bash
-root@ip-172-20-116-230:~# docker stop $(docker ps | grep "k8s.gcr.io/etcd" | awk '{print $1}')
-root@ip-172-20-116-230:~# rm -r /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/member/
-root@ip-172-20-116-230:~# rm -r /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/member/
-```
-
-Launch them again:
-
-```bash
-root@ip-172-20-116-230:~# systemctl start kubelet
-```
+### Update Cluster to launch new masters
 
-At this point, both etcd clusters should be healthy with two members:
+Update the cluster spec and apply the config by running the following:
 
 ```bash
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member list
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl cluster-health
-$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member list
-$ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 cluster-health
+kops replace -f mycluster.yaml
+kops update cluster example.com
+kops update cluster example.com --yes
 ```
 
-If not, check ``/var/log/etcd.log`` for problems.
+This will launch the two new masters. You will also need to roll the old master so that it can join the new etcd cluster.
 
-Restart protokube on the new master:
+After about 5 minutes all three masters should have found each other. Run the following to ensure everything is running as expected.
 
 ```bash
-root@ip-172-20-116-230:~# systemctl start protokube
+kops validate cluster
 ```
 
-## 4 - Add the third master
-
-### a - Edit instance group
-
-Prepare to launch the third master instance:
+While rotating the original master is not strictly necessary, kops will say it needs updating because of the configuration change.
 
-```bash
-$ kops edit instancegroup master-<availability-zone3>
 ```
-
-* Replace ``maxSize`` and ``minSize`` values to **1**.
-
-### b - Add a new member to the etcd clusters
-
- ```bash
- $ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-<availability-zone3-name> http://etcd-<availability-zone3-name>.internal.example.com:2380 \
-	&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-<availability-zone3-name> http://etcd-events-<availability-zone3-name>.internal.example.com:2381
- ```
-
-Example:
-
-```bash
-$ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member add etcd-c http://etcd-c.internal.example.com:2380 \
-	&& kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member add etcd-events-c http://etcd-events-c.internal.example.com:2381
-```
-
-### c - Launch the third master
-
- ```bash
- $ kops update cluster example.com --yes
- # wait for the third master to boot and initialize
- $ ssh admin@<third-master>
- admin@ip-172-20-139-130:~$ sudo -i
- root@ip-172-20-139-130:~# systemctl stop kubelet
- root@ip-172-20-139-130:~# systemctl stop protokube
- ```
-
- Reinitialize the etcd instances:
- * In both ``/etc/kubernetes/manifests/etcd-events.manifest`` and
- ``/etc/kubernetes/manifests/etcd.manifest``, edit the
- ``ETCD_INITIAL_CLUSTER_STATE`` variable to ``existing``.
- * Delete the containers and the data directories. Mount paths can be determined with the help of ``docker inspect <container-id> | grep /mnt/master-vol``:
-
- ```bash
- root@ip-172-20-139-130:~# docker stop $(docker ps | grep "k8s.gcr.io/etcd" | awk '{print $1}')
- root@ip-172-20-139-130:~# rm -r /mnt/master-vol-019796c3511a91b4f/var/etcd/data-events/member/
- root@ip-172-20-139-130:~# rm -r /mnt/master-vol-0c89fd6f6a256b686/var/etcd/data/member/
- ```
-
- Launch them again:
-
- ```bash
- root@ip-172-20-139-130:~# systemctl start kubelet
- ```
-
- At this point, both etcd clusters should be healthy with three members:
-
- ```bash
- $ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl member list
- $ kubectl --namespace=kube-system exec etcd-server-ip-172-20-36-161.ec2.internal -- etcdctl cluster-health
- $ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 member list
- $ kubectl --namespace=kube-system exec etcd-server-events-ip-172-20-36-161.ec2.internal -- etcdctl --endpoint http://127.0.0.1:4002 cluster-health
- ```
-
- If not, check ``/var/log/etcd.log`` for problems.
-
- Restart protokube on the third master:
-
- ```bash
- root@ip-172-20-139-130:~# systemctl start protokube
- ```
-
-## 5 - Cleanup
-
-To be sure that everything runs smoothly and is setup correctly, it is advised
-to terminate the masters one after the other (always keeping 2 of them up and
-running). They will be restarted with a clean config and should join the others
-without any problems.
-
-While optional, this last step allows you to be sure that your masters are
-fully configured by Kops and that there is no residual manual configuration.
-If there is any configuration problem, they will be detected during this step
-and not during a future upgrade or, worse, during a master failure.
-
-
-## 6 - Restore (if migration to multi-master failed)
-
-In case you failed to upgrade to multi-master you will need to restore from the backup you have taken previously.
-
-Take extra care because kops will not start etcd and etcd-events with the same ID on <master-b> an/or <master-c> for example but will mix them (ex: etcd-b and etcd-events-c on <master-b> & etcd-c and etcd-events-b on <master-c> ); this can be double checked in Route53 where kops will create DNS records for your services.
-
-If your 2nd spinned master failed and cluster becomes inconsistent edit the corresponding kops master instancegroup and switch ``MinSize`` and ``MaxSize`` to "0" and run an update on your cluster.
-
-Next ssh into your primary master:
-
-``systemctl stop kubelet``
-``systemctl stop protokube``
-
-Reinitialize the etcd instances:
-* In both ``/etc/kubernetes/manifests/etcd-events.manifest`` and
-``/etc/kubernetes/manifests/etcd.manifest``, add the
-``ETCD_FORCE_NEW_CLUSTER`` variable with value ``1``.
-* Delete the containers and the data directories while restoring also from previous backup:
-
-```bash
-root@ip-172-20-116-230:~# docker stop $(docker ps | grep "k8s.gcr.io/etcd" | awk '{print $1}')
-root@ip-172-20-116-230:~# rm -r /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/member/
-root@ip-172-20-116-230:~# rm -r /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/member/
-root@ip-172-20-116-230:~# cp -R /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/backup/member  /mnt/master-vol-03b97b1249caf379a/var/etcd/data-events/
-root@ip-172-20-116-230:~# cp -R /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/backup/member /mnt/master-vol-0dbfd1f3c60b8c509/var/etcd/data/
-```
-
-Now start back the services and watch for the logs:
-
-``systemctl start kubelet``
-``tail -f /var/log/etcd*`` # for errors, if no errors encountered re-start also protokube
-``systemctl start protokube``
-
-Test if your master is reboot-proof:
-
-Go to EC2 and ``Terminate`` the instance and check if your cluster recovers (needed to discard any manual configurations and check that kops handles everything the right way).
-
-Note! Would recommend also to use Amazon Lambda to take daily Snapshots of all your persistent volume so you can have from what to recover in case of failures.
\ No newline at end of file
+kops rolling-update cluster --yes
+```
\ No newline at end of file
diff --git a/pkg/apis/kops/validation/BUILD.bazel b/pkg/apis/kops/validation/BUILD.bazel
index fd4795343a..d8a29e0a30 100644
--- a/pkg/apis/kops/validation/BUILD.bazel
+++ b/pkg/apis/kops/validation/BUILD.bazel
@@ -38,6 +38,7 @@ go_test(
     name = "go_default_test",
     srcs = [
         "aws_test.go",
+        "cluster_test.go",
         "instancegroup_test.go",
         "validation_test.go",
     ],
diff --git a/pkg/apis/kops/validation/cluster.go b/pkg/apis/kops/validation/cluster.go
index 6ab2797c50..d8e6ab4441 100644
--- a/pkg/apis/kops/validation/cluster.go
+++ b/pkg/apis/kops/validation/cluster.go
@@ -88,9 +88,7 @@ func validateEtcdClusterUpdate(fp *field.Path, obj *kops.EtcdClusterSpec, status
 			fp := fp.Child("etcdMembers").Key(k)
 
 			oldMember := oldMembers[k]
-			if oldMember == nil {
-				allErrs = append(allErrs, field.Forbidden(fp, "EtcdCluster members cannot be added"))
-			} else {
+			if oldMember != nil {
 				allErrs = append(allErrs, validateEtcdMemberUpdate(fp, newMember, etcdClusterStatus, oldMember)...)
 			}
 		}


Test patch:
diff --git a/pkg/apis/kops/validation/cluster_test.go b/pkg/apis/kops/validation/cluster_test.go
new file mode 100644
index 0000000000..98b16d4a2d
--- /dev/null
+++ b/pkg/apis/kops/validation/cluster_test.go
@@ -0,0 +1,113 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package validation
+
+import (
+	"testing"
+
+	"k8s.io/kops/upup/pkg/fi"
+
+	"k8s.io/kops/pkg/apis/kops"
+)
+
+func TestValidEtcdChanges(t *testing.T) {
+	grid := []struct {
+		OldSpec *kops.EtcdClusterSpec
+		NewSpec *kops.EtcdClusterSpec
+		Status  *kops.ClusterStatus
+		Details string
+	}{
+		{
+			OldSpec: &kops.EtcdClusterSpec{
+				Name: "main",
+				Members: []*kops.EtcdMemberSpec{
+					{
+						Name:          "a",
+						InstanceGroup: fi.String("eu-central-1a"),
+					},
+				},
+			},
+
+			NewSpec: &kops.EtcdClusterSpec{
+				Name: "main",
+				Members: []*kops.EtcdMemberSpec{
+					{
+						Name:          "a",
+						InstanceGroup: fi.String("eu-central-1a"),
+					},
+					{
+						Name:          "b",
+						InstanceGroup: fi.String("eu-central-1b"),
+					},
+					{
+						Name:          "c",
+						InstanceGroup: fi.String("eu-central-1c"),
+					},
+				},
+			},
+
+			Status: &kops.ClusterStatus{
+				EtcdClusters: []kops.EtcdClusterStatus{
+					{
+						Name: "main",
+					},
+				},
+			},
+
+			Details: "Could not update from single to multi-master",
+		},
+
+		{
+			OldSpec: &kops.EtcdClusterSpec{
+				Name: "main",
+				Members: []*kops.EtcdMemberSpec{
+					{
+						Name:          "a",
+						InstanceGroup: fi.String("eu-central-1a"),
+					},
+				},
+			},
+
+			NewSpec: &kops.EtcdClusterSpec{
+				Name: "main",
+				Members: []*kops.EtcdMemberSpec{
+					{
+						Name:          "a",
+						InstanceGroup: fi.String("eu-central-1a"),
+					},
+				},
+			},
+
+			Status: &kops.ClusterStatus{
+				EtcdClusters: []kops.EtcdClusterStatus{
+					{
+						Name: "main",
+					},
+				},
+			},
+
+			Details: "Could not update identical specs",
+		},
+	}
+
+	for _, g := range grid {
+		errorList := validateEtcdClusterUpdate(nil, g.NewSpec, g.Status, g.OldSpec)
+		if len(errorList) != 0 {
+			t.Error(g.Details)
+		}
+	}
+}