RCA: Elevated git error rate on 2019-07-18

## Summary

Increased error rates for ~"Service:Gitaly" due to large file uploads by a single user

- Service(s) affected : ~"Service:Gitaly"  
- Team attribution :  
- Minutes downtime or degradation : 23 (1642-1704UTC)

For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/ZUei7TkWz/platform-metrics?orgId=1) to look at appdex and SLO violations.  

![image](/uploads/685ee0366e1cdcf876fa4c8b84ba485c/image.png)
![image](/uploads/1046c5ce826f5ab008f4150a1bb87e42/image.png)

## Impact & Metrics

Start with the following:

- What was the impact of the incident? partial service degradation  
- Who was impacted by this incident? external customers  
- How did the incident impact customers? (i.e. preventing them from doing X, incorrect display of Y, ...)
- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.


## Detection & Response

Start with the following:

- How was the incident detected? alertmanager/pagerduty notification
- Did alarming work as expected? yes
- How long did it take from the start of the incident to its detection? Approx. 5 minutes
- How long did it take from detection to remediation? Approx. 15 minutes
- Were there any issues with the response to the incident? No

## Timeline

2019-07-18

- 16:46 UTC - received alerts of increased error rates for Gitaly on `file-23-stor-gprd`
- 16:56 UTC - confirmed that the errors were due to operations from the user identified in production#972
- 17:01 UTC - EOC initiated a user block
- 17:01 UTC - Load average on `file-23-stor-gprd` begins dropping
- 17:01 UTC - Alert `Gitaly error rate is too high` [cleared](https://gitlab.pagerduty.com/incidents/PD8YZ3Q)
- 17:06 UTC - Node-level alert `Gitaly error rate is too high` for `file-23-stor-gprd` [cleared](https://gitlab.pagerduty.com/incidents/PZZNYR0)

~~At first glance, this is likely~~ This was a duplicate/recurrence of production#972

## Root Cause Analysis

Gitaly service on `file-23-stor-gprd.c.gitlab-production.internal` was unresponsive

1. Why? - Gitaly was consuming 100% CPU on `file-23-stor-gprd`  
1. Why? - Gitaly `SSHReceivePack` and `SSHUploadPack` processes servicing requests were [consuming all CPU resources](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7281#note_193666706)  
1. Why? - A user attempted to upload very large file(s) (>=10GB) via SSH  

## What went well

1. Alerting allowed us to identify the root problem quickly
1. On-call engineers worked well across rotations to handle/respond to multiple incidents
1. Support was engaged to interface with the affected customer and request that they configure `git-lfs` for better handling of large files

## What can be improved

1. We need to better understand why operations from a single user were able to consume all of the resources for a single Gitaly shard


## Corrective actions

1. Do we already have / can we implement a facility for Gitaly to abort git operations that take longer than some time threshold - #7388  
1. Investigate using resource constraint mechanism (like cgroups) to limit the resources that any one git operation can consume - #7387  

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 2.0