High-level planning and discussion for OS upgrades (16.04 to 20.04)

The following is an initial _**rough draft**_ of pros and cons for each method. I'm sure there are many more pros and cons, but I'd like to open this up for input while I continue to consider the options. 

This list will eventually be used to determine how to proceed. We should mainly consider patroni first, but I included others as I thought of them.

# Pros and Cons of Upgrade Methods

I am only considering pet services such as patroni and gitaly. Other nodes such as git and web are much more ephemeral and are likely going to be easiest to just rebuild.
Patroni is in a high availability configuration and thus we could do the upgrades with only one bit of downtime during the failover of the primary. Gitaly nodes on the other
hand are single points of failure and are unlikely to be able to be upgraded without downtime.

## Delete and rebuild server with 20.04

I would expect this process to be done via terraform by updating the base image for a given module and applying it in a targeted, one by one, fashion.
The disk would not be deleted while applying the terraform and would be reattached as-is to the newly built server.

This is my preferred method for redis, patroni, and ideally gitaly.

### Pros

- We will have a completely fresh server
- Unlikely to end up in a state where a server doesn't come back from boot
- Will be able to prove that this is a viable process for stateful servers in the future
- By rebuilding these we could also implement the required changes for GSM (https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/343)
- An excellent time to upgrade machine types as well (https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12484#note_505538821, https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/12294)

### Cons

- A rebuild might take some time to bootstrap and become available
- Gitaly is a single point of failure, so doing this there might take more downtime than other methods
- Terraform would be "dirty" for however long it takes to go through and rebuild each server
  - There are around 60 gitaly servers so this would probably be quite a while
- It is always unnerving to delete servers (not something we should base a decision on, simply wanted to express :slight_smile:)

## In place update to 20.04

With this path, we would run an in place upgrade in which the only time the node would be down would be for the reboot.
We could set ignore_changes in the storage terraform module to allow new servers to be built with new images while not trying to force rebuilds on the old ones.

This is my preferred method for CI nodes s. 

### Pros

- Doesn't require rebuild of server
- Shorter downtime for a service like Gitaly as the downtime would only be a reboot
- Would avoid rebuilding CI nodes since they are all fairly manual setup

### Cons

- Potential for strange behavior if the upgrade goes awry for some reason
- If upgrade does go awry, recovering from that may be difficult and would likely be faster to fallback to the rebuild plan
- We could update the machine type at the same time, but it would be a bit more difficult than rebuilding as it would involve terraform changes and applies unrelated to the upgrade.
sd
# Gitaly specific options

## Subscribe to Ubuntu Advantage

This option would have us subscribe to [Advantage](https://ubuntu.com/advantage/) which would give us software and security updates until up to 2024. There would be a yearly fee associated with this option. A subscription for 80 VMs would be 6k a year with no support. If we wanted support it would be 20k or 40k per year based on whether we wanted 5 or 7 day access to support respectively. This would give us software and security updates until 2024, which ideally will be plenty of time. This in conjunction with ignoring boot disk and image changes in the storage module would allow us to build new servers using a new image but the same process.

This is my preferred method for Gitaly.

### Pros

- We can have quite a bit of runway if necessary to try to figure out the best way to update these, potentially with zero downtime.
- We could thus focus on all of the other nodes we need to upgrade this quarter
- The cost isn't really that expensive.

### Cons

- It costs. It is reasonable, but it does still cost.
- We would be further kicking the can down the road (which is really only a minor downside given the benefit of extra time).
- We're wanting to stop building GitLab packages for 16.04 so this would be a problem.

## Upgrade by switching over to a newly built server

This method would involve building an additional server running 20.04, configuring it as desired, and then switching the data disk between the old and new nodes.
These servers would likely come in the form of a new terraform module definition.

I do not think this a particularly great idea for any of our cases. I believe the only reasonable use of this method would be for Gitaly.

### Pros

- We could likely do the gitaly updates with the shortest amount of downtime this way, if we were able to solve the problems below.

### Cons

- We would have to have some method to switch traffic from the old node to the new node
  - Currently all servers are referenced by DNS name which would be different with new servers
  - The DNS name cannot be changed once the server is created
  - Rolling out config changes to update the location of the gitaly shards is going to be a long and painful process, with high risk
  - If we wanted to try to mitigate risk, we could create some sort of haproxy system in order to be able to swap traffic between the nodes
- This method would likely be a substantial amount of extra work in validating everything as well as planning and executing all the switchovers

## Build new servers and migrate all the repos from one node to the new node

This method would involve using the repo migration api to move all repos from one gitaly shard to a new one. I do not recommend this method.

### Pros

- We could upgrade the gitaly nodes with no downtime

### Cons

- The migration API is good, but there would undoubtedly be a decent number of failures, each of which would require investigation, validation, and repair
- This would take ages. The migration of repos between shards is a slow process and we have about 60 shards to go through

Story points: 8.0