RCA for 2018-12-21 Gitaly Outage

**Please note:** if the incident relates to sensitive data, or is security related consider
labeling this issue with ~security and mark it confidential.
***

## Summary

A brief summary of what happened. Try to make it as executive-friendly as possible.

1. Service(s) affected : Gitaly Storage Nodes
2. Team attribution : Infrastructure
3. Minutes downtime or degradation : 35 Minutes 

## Impact & Metrics

Start with the following:

- What was the impact of the incident? All customer and internal requests to Git data nodes were unable to be serviced for 34 minutes.
- Who was impacted by this incident? External Customers, CI jobs
- How did the incident impact customers? See impact above
- How many attempts were made to access the impacted service/feature? 
- How many customers were affected? All
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.


## Detection & Response

Start with the following:

- How was the incident detected? PagerDuty/Slack Alerts
- Did alarming work as expected? Yes
- How long did it take from the start of the incident to its detection? Approximately 10 minutes
- How long did it take from detection to remediation? 30 minutes
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)
Yes - terraform module repos that were only on .com delayed us in running terraform to stand the compute node back up until a local copy was used.

## Timeline

On the production issue.

## Root Cause Analysis

While working on our Disaster Recovery project and region, an SRE on our team was unable to use Terraform to properly remove some nodes in the the DR project and region.  They chose to go to the GCP console to perform the deletes to get the Terraform state back to good.  While doing this, they searched for some nodes and per the illustration below had their project switched from gitlab-dr to gitlab-production.  It was not clear that the project switch had been made and they proceeded to remove the gitaly compute instances (file-[1-24]) in the gitlab-production project.  At that point, monitoring started to alert us to the problem and the team started to restore the deleted compute nodes.

### Illustration

When attempting to search for something in the GCP search bar which has a partial match, one would expect that pressing enter here would execute a search for a partial IP address, which in this case returns no results.

![Search](/uploads/e2f6e884da37a81f47cd38613abfffba/Search.png)

Pressing enter results in this:

![Changed](/uploads/48988c5767305f64a109f595f342a560/Changed.png)

Which appears to have searched and found nothing.  However, it did not search - the first line in the dropdown above was highlighted - so the project changed. At this point, going and deleting the DR file nodes resulted in the gitlab-production nodes being deleted rather than the gitlab-dr nodes.

## What went well

Start with the following:

- Identify the things that worked well or as expected.
- Any additional call-outs for what went particularly well.

1.  Quickness of team to jump on a zoom and start to mitigate the issue.
1.  We were able to restore the affected infrastructure with no data loss

## What can be improved

Start with the following:

- Using the root cause analysis, explain what can be improved to prevent this from happening again.
- Is there anything that could have been done to improve the detection or time to detection?
- Is there anything that could have been done to improve the response or time to response?
- Is there an existing issue that would have either prevented this incident or reduced the impact?
- Did we have any indication or beforehand knowledge that this incident might take place?

1. Look at different node names for disaster recovery compute and storage node names vs production. 
2. Mirror all Terraform repos (environments and modules) on ops.gitlab.net to prevent issues with access when GitLab.com is down.
3. Look at further enhancing procedures for any deletes in production requiring two sets of eyes and ways to prevent needing to do any interaction with the cloud console.  Further automation to ask and double check before performing the delete.


## Corrective actions

1. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5815 : Add deletion protection for gitaly servers
2. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5816 : Move terraform modules to the ops instance
3. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5868 : Start practicing incident response
4. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5867 : Create list of incident response scenarios
5. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5869 : Setup Atlantis for Terraform deployments on ops instance
6. https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5945 : Change to TF process to prevent conflicts that sent us to Cloud console

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 3.0