opened
Shrugs
username_0
We wanted to migrate from a kube-up-created 1.2.x cluster to a kops-managed 1.4.0 cluster with 0 downtime. After reading the docs in [upgrade_from_k8s_12.md](https://github.com/kubernetes/kops/blob/master/docs/upgrade_from_k8s_12.md), we decided to go with the following strategy to avoid the possibility of downtime. The following steps are taken from the writeup I did internally to document our approach.

1. Delegate cluster-level dns resolution to Route53 by adding appropriate NS records pointing `cluster.example.com` to Route53's Hosted Zone's nameservers.
2. Create the new cluster's configuration files with kops
  - `kops create cluster --cloud=aws --zones=us-east-1a,us-east-1b --admin-access=12.34.56.78/32 --dns-zone=cluster.example.com --kubernetes-version=1.4.0 --node-count=14 --node-size=c3.xlarge --master-zones=us-east-1a --master-size=m4.large --vpc=vpc-123abcdef --network-cidr=172.20.0.0/16 cluster.example.com`
  - note that the `--network-cidr` is the cidr of the existing VPC.
  - Note that `kops` will propose re-naming the existing VPC but the change never occurs. We eventually manually changed it to the correct value.
4. Verify that the CIDR on each of the zone subnets does not overlap with an existing subnet's.
5. Verify the planned changes with `kops update cluster cluster.example.com`
5. Create the cluster with `kops update cluster cluster.example.com --yes`
6. Wait around for the cluster to fully come up and be available. `k get nodes` should return `(master + minions) = 15` available nodes.
7. (Optional) Create the Dashboard with `kubectl create -f https://rawgit.com/kubernetes/dashboard/master/src/deploy/kubernetes-dashboard.yaml`
8. Add the generated `nodes.cluster.example.com` security group to the resources that may need it (i.e. ElastiCache, RDS, etc)
9. Deploy the same resource configuration to the new cluster.
10. Wait patiently while everything happens.
11. Test that everything "just works" by hitting the ELB's dns entry directly; you should be able to use the site as expected.
13. To transition traffic from the old cluster to the new cluster, update the `CNAME` record for `example.com` to point to the new ELB's DNS name.
  - note that if you're proxying through cloudflare, changes are instantaneous because it's technically a reverse proxy and not a DNS change.
  - if not using cloudflare or another reverse proxy solution, you'll want to update you DNS record's TTL to a very low duration about 48 hours in advance of this change (and then change it back to the previous value once the shift has been finalized).
14. Rejoice.
15. Once traffic has shifted from the old cluster (relatively quickly), delete the old resources after confirming that traffic has stabilized and that no new errors are generated.

## Resources

This is a list of resource the old cluster is using that should be deleted. This may not be complete.

- autoscaling group (`kubernetes-minion-group-us-east-1a`)
- launch config (`kubernetes-minion-group-us-east-1a` and `kubernetes-minion-group-us-east-1aCopy`)
- all ec2 instances (`tag:KubernetesCluster : kubernetes`)
- all associated EBS volumes (most likely released when the instances are terminated)
- security groups (`tag:KubernetesCluster : kubernetes`)

## Recovery/Rollback

The only part of this procedure that should affect the users actively using the site is the DNS swap, which should be relatively instantaneous because we're using Cloudflare as a reverse proxy, not just as a nameserver. In the event that we need to swap back to the old cluster, we can replace the example.com CNAME record with the original ELB's DNS name.


---

I'm willing to write up a PR to add this to the document about migrations from previous clusters.

Are there any downsides we can surface around this approach? I believe kube-up also uses the route table, so we're unable to use this strategy for a cluster that requires `>25` nodes (since 50 is the artificial limitation on VPC Route Tables). If you're able to gradually transition traffic from one cluster to another (perhaps another load balancer to split traffic between clusters) you could slowly scale the new cluster up and the old cluster down to avoid hitting the 50-entry cap on the route table. Not an ideal situation, though.
[docs] Discussing alternative cluster migration strategies
issue
created
chrislovecnm
username_1
@justinsb what do you think?
comment
created
Shrugs
username_0
Bumping. If this is something we want to add to the https://github.com/kubernetes/kops/blob/master/docs/upgrade_from_k8s_12.md doc, I can make a PR.
comment
created
chrislovecnm
username_1
@username_0 been at kubecon 😀 

Actually, the name of that doc sucks so rename it.  The content is great, make it AWESOME.  When have I turned down a PR?  You the man.
comment
closed
chrislovecnm
username_1
issue