RCA for June 2nd GCP related GitLab.com incident

@ahanselka and @ahmadsherif - would you be okay owning this RCA?  I'll add questions on my side too.
cc @andrewn 

## Summary

Google had a major networking outage in their US East regions that affected their entire infrastructure, including GitLab.com

- Service(s) affected : All services for GitLab.com
- Team attribution : External
- Minutes downtime or degradation : 20 minutes of downtime, 190 total of degradation

https://dashboards.gitlab.net/d/ZUei7TkWz/platform-metrics?orgId=1&fullscreen&panelId=3&from=1559498453907&to=1559516462767

## Impact & Metrics

Start with the following:

- All GitLab.com services were completely down for about 30 minutes as Postgres failed over and services had to be restarted/HUPed. The application had degradation for about 3.5 hours after that.
- All users of GitLab.com were affected by this incident.
- During the first 30 minutes, GitLab.com was entirely inaccessible to users. For the remainder of the incident there was an elevated rate of 5xx errors.
  - The main reason GitLab.com went completely down is due to Patroni cluster instability with failovers and the application being unable to follow the new primary.
  - The longer period of elevated rate of 5xx errors was related to the network instability.
- Many of our alerts were flapping because the monitoring server was unable to reach the servers in question.
- There was a lot of flapping between redis-cache primary and secondaries. This did not seem to have an affect on the availability of the application.

![patroni-network-drops](/uploads/623daf2bd0b5dd18bafed0a0c811855b/patroni-network-drops.png)

![error-rates-and-resolution](/uploads/4a69fb6adf9c14137b0d0e0500d22e76/error-rates-and-resolution.png)

## Detection & Response

Start with the following:

- The incident was detected via PagerDuty alerts for GitLab.com being down.
- Alarming worked as expected, however there were so many alerts it was overwhelming and made it hard to quickly determine anything.
- Because of the deluge of alerts, some important relevant alerts such as the alert indicating Postgres failed over were lost in the noise.
- It took about 2-3 minutes after the beginning of the downtime to get alerted and begin response.
- It took about 30 minutes for us to recover GitLab.com from the Postgres failover, however the site remained unstable due to the provider outage.
- Our dashboards were partially broken, which was a known issue earlier in the week (https://gitlab.com/gitlab-com/gl-infra/production/issues/849), making it much more difficult to get started with the response.

## Timeline

2019-06-02

* 18:50 UTC - Patroni failed over from -04 to -06
* 19:05 UTC - Most of our Grafana dashboards are inconsistently working because of thanos issues: https://gitlab.com/gitlab-com/gl-infra/production/issues/849
* 19:07 UTC - Pingdom returning errors
* 19:12 UTC - Diagnosis of Postgres failover
* 19:21 UTC - Services hup'd
* 19:26 UTC - GitLab.com operational. Pingdom reporting services as up.
* 19:41 UTC - Watching https://status.cloud.google.com/incident/compute/19003
* 19:59 UTC - also watching https://status.cloud.google.com/incident/cloud-networking/19009
* 20:39 UTC - Continuing to monitor google incidents
* 21:22 UTC - another failover from patroni-04 to -01
* 21:54 UTC - postgres failed back over to -04
* 22:00 UTC - Error rates returned to normal (that is, there were none)

2019-06-03

* 09:40 UTC - Restoring tuple statistics by running cluster-wide `ANALYZE`, see https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5841#note_128321668 (done 10:10 UTC)

2019-06-06

* Google posted [their RCA](https://status.cloud.google.com/incident/cloud-networking/19009)


## Root Cause Analysis

GitLab.com went down for 30 minutes with instability over the course of 4 total hours.

1. Why? - The application could no longer reach the database.
2. Why? - Postgres failed over unexpectedly.
3. Why? - There was networking instability which caused the cluster to try to fail over multiple times.
4. Why? - GCP had a major networking outage in the east region where we are located.

## What went well

Start with the following:

- We found out very quickly that there was a problem
- Multiple people jumped in to help diagnose and repair the issue
- Delegation of duties and expectations was clear and effective
  - i.e. "You go update the status page", "I will go restart sidekiq", etc.
- Patroni failover was successful. If we had still been on Repmgr it would have been true disaster.

## What can be improved

Start with the following:

- We should better automate and tune our database failover process so that the application can gracefully handle failovers.
  - We could try to execute more failovers in staging and eventually production to be more confident that a similar incident in the future would not cause a complete outage as it did in this case.
- We should also automate such that we don't have to re-run `ANALYZE` on the tables to re-populate statistics. 
- We can try to prune and curate our alerts such that there isn't a massive deluge of alerts that obscure the problem and make it hard to see other relevant alerts.
- While this did not directly affect production, we didn't notice that staging had fallen apart also as a result of this. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6854. It wasn't important to fix ASAP on a Sunday evening, but we should create a follow-up issue immediately when there is an issue like this so someone can follow up on Monday.


## Corrective actions

Some of these issues are not created as a reaction specifically to this incident but are the correct actions.

- Automated failover testing (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5890)
- Clients still connect to old primary after failover (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5675)
- Graceful Patroni failovers (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5833)
- Production incident relating to Thanos problems (https://gitlab.com/gitlab-com/gl-infra/production/issues/849)
- Use a virtual IP for failovers (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7059)


## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 4.0