RCA: High Rails Error Rate on Front End

Received alerts for elevated error rates: `Rails is returning 5xx errors at a high rate for git . Traffic is impacted and users are likely seeing 500 errors.`

Service(s) affected : ~"Service:GitLab Rails" ~"Service:Git" ~"Service:API" ~"Service:Web"  
Team attribution : @gitlab-com/gl-infra  
Minutes downtime or degradation : 178 minutes

## Impact & Metrics

Start with the following:

- What was the impact of the incident? portions of the api, web, and git fleet were not registered with the load-balancers, resulting in increased 5xx error rates for those services
- Who was impacted by this incident? GitLab.com users / external customers
- How did the incident impact customers? 5xx errors attempting to make API calls, or perform git operations via HTTPS
- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.

## Detection & Response

Start with the following:

- How was the incident detected? on-call engineer received pagerduty alert(s)  
- Did alarming work as expected? yes, though some alerts were ignored initially due to concurrence with ongoing deployment
- How long did it take from the start of the incident to its detection?
- How long did it take from detection to remediation?
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)

## Timeline

2019-06-11

* 22:55 UTC - deployment started for hotpatch https://ops.gitlab.net/gitlab-com/gl-infra/patcher/merge_requests/100
* 23:03 UTC - deployment fails, engineer experiences issues adding/removing backend servers from haproxy
* 23:40 UTC - deployment restarted
* 23:44 UTC - first evidence in grafana of imbalanced distribution for connections to https_git servers

2019-06-12

* 00:16 UTC - first instance of alert fires for increased error rate
* 01:06 UTC - on-call engineer alerted of increased error rates
* 01:09 UTC - alert acknowledged by on-call engineer
* 01:11 UTC - alert auto-resolves
* 01:19 UTC - alert recurs
* 01:24 UTC - alert auto-resolves
* 01:28 UTC - alert recurs
* 01:28 UTC - alert acknowledged by on-call engineer
* 01:31 UTC - investigation started
* 01:32 UTC - inconsistent load distribution identified as probable cause
* 01:35 UTC - IMOC notified via slack
* 01:45 UTC - IMOC paged via slack
* 01:57 UTC - impact to other backends identified
* 01:59 UTC - team begins working to register all affected backend servers with haproxy load-balancers
* 02:35 UTC - git service fully restored
* 02:42 UTC - api and web services restored

## Root Cause Analysis

### 5-whys
The front-end fleet experienced elevated error rates and disruption of service.  

1. Why? - portions of the api, web, and git fleet were not registered with the load-balancers
2. Why? - they were not re-registered properly during/after a deployment
3. Why? - the first hot-patch deployment pipeline was canceled
4. Why? - the pipeline appeared hung
5. Why? - the script incorrectly detected nodes' status

## What went well

Start with the following:

- Identify the things that worked well or as expected.
- Any additional call-outs for what went particularly well.

## What can be improved

Start with the following:

- Using the root cause analysis, explain what can be improved to prevent this from happening again.
- Is there anything that could have been done to improve the detection or time to detection?
- Is there anything that could have been done to improve the response or time to response?
- Is there an existing issue that would have either prevented this incident or reduced the impact?
- Did we have any indication or beforehand knowledge that this incident might take place?


## Corrective actions

- List issues that have been created as corrective actions from this incident.
- For each issue, include the following:
    - <Bare Issue link> - Issue labeled as ~"corrective action".
    - Include an estimated date of completion of the corrective action.
    - Incldue the named individual who owns the delivery of the corrective action.


## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 2.0