RCA For 2019-04-24 Google Load Balancer Anomalies

## Summary

There was a problem with Google ILB health checks which was causing the database load balancer to send traffic to read-only secondary nodes instead of the primary.

The Incident Issue is: https://gitlab.com/gitlab-com/gl-infra/production/issues/802

The problem started out slowly in the afternoon, at first mostly manifesting in the Pull Mirrors failing, but over the course of several hours it started compounding as other sidekiq jobs started retrying and web nodes started being directed to the read only databases.


Service(s) affected : Database and therefore everything in the us-east1-d zone

Team attribution : 

Minutes downtime or degradation : 

## Impact & Metrics


- What was the impact of the incident? 
Increased errors, especially in sidekiq jobs. Pull mirrors didn't work

- Who was impacted by this incident? 
All gitlab.com users, especially those using pull mirrors

- How did the incident impact customers? 
Mirrors were unable to run and sidekiq jobs failed. Web hooks failed.

- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.


## Detection & Response


- How was the incident detected?  
The first alert was for pull mirrors overdue. There was very little actual evidence in the first hour or so, only seemingly unconnected suspicious behavior.  The incident issue was opened based on instinct and suspicion rather than data.

- Did alarming work as expected?  
The existing alerts worked based on the number of errors observed, but could have been more helpful with identifying the source of the problem.

- How long did it take from the start of the incident to its detection?  
25 minutes

- How long did it take from detection to remediation?  
14 hours to full remediation

- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)

The Rackspace support layer adds a lot of overhead between us and google: https://gitlab.com/gitlab-com/gl-infra/production/issues/802#note_163652594



## Timeline

2019-04-24

- 00:00 UTC - Pull Mirror Jobs stopped processing
- 00:25 UTC - First alert to pager
- 00:30 UTC - Troubleshooting....
- 01:42 UTC - Ran last resort commands from runbook, changed, but not fixed
- 03:00 UTC - [Incident created](https://gitlab.slack.com/archives/CB7P5CJS1/p1556074810015200)<br><br>
troubleshooting continued...<br>
- 06:25 UTC - Incident call started<br><br>
- 06:40 UTC - Tweet sent out and updated status.io
- 06:50 UTC - Rackspace ticket opened: https://portal.rackspace.com/1173105/tickets/details/190424-ord-0000235
- 07:00 UTC - First attempt to call Rackspace
- 07:01 UTC - Paged DBRE for assessment 
- 07:40 UTC - Got a hold off Rackspace team and hopped on a Zoom to troubleshoot the issue on our end
- 07:52 UTC - GCP responded confirming small amount of traffic was sent to unintentional nodes
- 08:46 UTC - Confirmed our theories that something was off with the LB
- 08:51 UTC - GCP responded with some more updates about narrowing down the issue
- 10:49 UTC - Removed all read-only patroni nodes from their respective instance groups except for the primary
- 11:18 UTC - Reset redis key set and triggered the workers to pick up jobs
- 11:36 UTC - Queued jobs to 0, alerts cleared up - tweeted and updated status.io
- 11:52 UTC - GCP still investigating
- 12:30 UTC - GCP resolved the inconsistency
- 15:33 UTC - Started sidekiq-cluster on the few sidekiq-* nodes in 1-d zone 

## Root Cause Analysis

An update from Google:

> What exactly was happening is that health status of ILB backends was failing to propagate to instances in us-east1-d, and thus requests from them were forwarded to all backends configured, not just to the healthy one (That is the reason why you observed traffic going to -b as well). ILB and as well as this issue is not happening close to the server VM side, but close to the client VM.

Last update from Google:

> After further investigation, we believe that the temporary inconsistency of ILB backend health observed by VM Instances in us-east1-d was caused by a rollout of one of our control plane components in that zone. This issue affected only small amount of Virtual Private Networks and is likely triggered only together with VPC Peering.
>
> The engineering team resolved the inconsistency and we believe there should be no further impact.


Follow the "**5 whys**" in a **blameless** manner as the core of the root-cause analysis.

1. Why? - Jobs are intermittently failing
2. Why? - They can't write to the database sometimes
3. Why? - The load balancer sends some traffic to the wrong databases
4. Why? - The load balancer is not interpreting health checks correctly
5. Why? - The rollout of a Google control plane component in us-east1-d caused health checks to be reported incorrectly to the load balancers

## What went well

Start with the following:

- Identify the things that worked well or as expected.

The team piled on this very nicely. Everyone contributed what they could and spread the workload out.

Even though we didn't have alerting that covered this case, the existing alerts let us know that something was 'just not right'

- Any additional call-outs for what went particularly well.

The team did an AMAZING job with some very difficult troubleshooting to narrow down an issue that didn't have much conclusive evidence pointing to the cause.<br><br>

Rackspace support was able to help us troubleshoot the issue to make sure we had all of our bases covered. Stackdriver - Monitoring Metrics was very helpful in checking the LB egress traffic and this was a good proof point that traffic was sent to unintended nodes/zones. The graphs exactly showed the impact and recovery very nice. <br><br>

Even in the midst of the chaotic incident, we took the time to create and execute peer-reviewed change request to remove instances from instance groups instead of taking actions on the fly. 

## What can be improved


- Whether this happens again is up to Google. We can put pressure on them to be more careful, and we can build some more resilience into some of our jobs - especially sidekiq jobs which aren't idempotent. 
- The incident started in the late afternoon Hawaii time. The US had left for the day and the EU had not yet woken up, so there was not a lot of help available for a few hours.
- The runbook for Pull Mirrors just says to post and ask for help in the `#backend` Slack channel. When this was posted, it was in the EU early morning hours, so it took a long time for anyone to see it.
- Our database alerts and monitoring don't cover problems like this.
- There was no indication beforehand that this might happen, no existing issue related to it, and no reason to think it was something we should plan for.
- We should brainstorm, plan and test scenarios like this to see how our system reacts. 
- We should practice making changes in our production infrastructure more often and for more scenarios so that we always feel comfortable doing similar changes during incident. During this incident, removing instances from instance groups VS removing instance groups from LB took a little bit of time because of uncertainty around the setup and how the system would react. We were able to collectively talk through, wrote down steps and executed - but this is something we should be able to practice even during non-incident time. 

## Corrective actions

- List issues that have been created as corrective actions from this incident.
- For each issue, include the following:
    - <Bare Issue link> - Issue labeled as ~"corrective action".
    - Include an estimated date of completion of the corrective action.
    - Include the named individual who owns the delivery of the corrective action.


## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 3.0