RCA Gitaly latency below SLO

<!--
**Please note:** if the incident relates to sensitive data, or is security related consider
labeling this issue with ~security and mark it confidential.
*** 
-->

Incident: gitlab-com/gl-infra/production#1013

## Summary

A file descriptor leak in gitaly on file-23 caused high resource consumption and thus very high latencies for all gitaly operations on this host.
This was causing high latencies for all projects on this host while all other projects stayed unaffected.
Some projects of severely blocked customers have been moved to another node during the investigation.

- Service(s) affected : ~"Service:Gitaly" 

- Team attribution : Gitaly

- Minutes downtime or degradation : 270

For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/ZUei7TkWz/platform-metrics?orgId=1) to look at appdex and SLO violations.

## Impact & Metrics

Start with the following:

- What was the impact of the incident?
  - Very high Gitaly latencies for projects hosted on file-23
- Who was impacted by this incident?
  - all customers with projects on file-23 (some big customers reported issues)
- How did the incident impact customers?
  - all gitaly based operations, like merge requests, have been very slow
- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.

Gitaly P99 latency
![image](/uploads/3535cbc05aa5452b6172c5be0458a4bc/image.png)

## Detection & Response

Start with the following:

- How was the incident detected?
  - Gitaly latency SLO alert
- Did alarming work as expected?
  - the Gitaly latency SLO alert did not go to pagerduty, so it wasn't seen by the EOC immediately
  - if we had alarming on gitaly file descriptor count, we could have seen issues days before (since 7/18)
- How long did it take from the start of the incident to its detection?
  - customers reported issues since 7/18 (the start of the fd leakage), so it took 13 days
- How long did it take from detection to remediation?
  - 258m
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)
  - while 503 issues on file-23 were [identified on 7/18 already](https://gitlab.com/gitlab-com/customer-success/account-triage/issues/119#note_194548970) we didn't take action

## Root Cause Analysis

Some projects suffered from high latencies for Gitaly-related operations (like MRs).

1. Why? - Because Gitaly had higher latencies for some projects.
2. Why? - Because Gitaly on file-23 had issues, which affected only projects hosted on that node.
3. Why? - Because the Gitaly process was using a lot CPU resources.
4. Why? - Because Gitaly had 10,000 hanging cat-file processes.
5. Why? - Because Gitaly was leaking file descriptors.

## What went well

Start with the following:

- Identify the things that worked well or as expected.
- Any additional call-outs for what went particularly well.

## What can be improved

- Alerting on gitaly file descriptors
- Make Gitaly SLO alerts go to pagerduty
- We should have followed up after we identified 503 errors on file-23 days ago
- We should have a clear escalation path from support to SRE



## Corrective actions

- List issues that have been created as corrective actions from this incident.
- For each issue, include the following:
    - <Bare Issue link> - Issue labeled as ~"corrective action".
    - Include an estimated date of completion of the corrective action.
    - Incldue the named individual who owns the delivery of the corrective action.

* [x] prevent cat-file leak https://gitlab.com/gitlab-org/gitaly/merge_requests/1390
* [x] alert on high gitaly file descriptor count https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7390
* [x] page for gitaly SLO alerts https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7391

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 3.0