RCA: Container Registry Deployment Deleted in the Production Kubernetes Cluster

## Summary

An engineer working locally was mistakenly connected to the production cluster.  The engineer sent a command intended for the local development cluster was instead sent to the production cluster which resulted in the deletion of the Kubernetes Deployment object for the Container Registry.  This brought down the service endpoint as no Pods were available and our HAProxy service started to send HTTP 503's otherwise known as "The server is currently unavailable" for any request bound to the Container Registry.

## Impact & Metrics

Start with the following:

- What was the impact of the incident? Container Service was down.
- Who was impacted by this incident? Anything that would have requested an image to/from the Container Registry (customers, CI jobs, etc)
- How did the incident impact customers? This prevents customers from uploading images and downloading images to the Container Registry
- How many attempts were made to access the impacted service/feature? As seen in the below chart, we sustained a request rate to the Container Registry at a rate above 150 requests per second
- How many customers were affected? TODO
- How many customers tried to access the impacted service/feature? TODO

Include any additional metrics that are of relevance.

![image](/uploads/c1c59190a24739854bbd406d638072c7/image.png)

[Source](https://dashboards.gitlab.net/d/oWe9aYxmk/pod-metrics?orgId=1&from=1568307753787&to=1568309759687&var-Deployment=gitlab-registry&var-env=gprd&var-cluster=gprd-gitlab-gke&var-Node=All&var-namespace=monitoring)

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.

![image](/uploads/d527063af64bca1baca2f88e77248327/image.png)

![image](/uploads/902dc64d520096d8abc9e82478801383/image.png)

[Source](https://dashboards.gitlab.net/d/AkOdlrSmk/imported-haproxy-stats?orgId=1&from=1568307581081&to=1568309405739)

Logs from Stackdriver: [console.cloud.google.com/logs](https://console.cloud.google.com/logs/viewer?project=gitlab-production&minLogLevel=0&expandAll=false&timestamp=2019-09-12T17%3A24%3A15.319215000Z&customFacets&limitCustomFacetWidth=true&dateRangeStart=2019-09-12T13%3A30%3A42.700Z&dateRangeEnd=2019-09-12T19%3A30%3A42.700Z&interval=PT6H&resource=k8s_cluster%2Flocation%2Fus-east1%2Fcluster_name%2Fgprd-gitlab-gke&scrollTimestamp=2019-09-12T17%3A24%3A15.180382000Z&filters=text%3Agitlab-registry&advancedFilter=resource.type%3D%22k8s_cluster%22%0Aresource.labels.location%3D%22us-east1%22%0Aresource.labels.cluster_name%3D%22gprd-gitlab-gke%22%0Atimestamp%3E%3D%222019-09-12T17%3A11%3A02.287936Z%22%0Atimestamp%3C%3D%222019-09-12T17%3A25%3A02.287936Z%22%0AprotoPayload.authenticationInfo.principalEmail%3D%22jskarbek%40gitlab.com%22%0AprotoPayload.resourceName%3D%22extensions%2Fv1beta1%2Fnamespaces%2Fgitlab%2Fdeployments%2Fgitlab-registry%22%20OR%20protoPayload.resourceName%3D%22apps%2Fv1beta2%2Fnamespaces%2Fgitlab%2Fdeployments%2Fgitlab-registry%22)

## Detection & Response

Start with the following:

- How was the incident detected? Alerts
- Did alarming work as expected? Yes
- How long did it take from the start of the incident to its detection? 12 minutes
- How long did it take from detection to remediation? 1 minute
- Were there any issues with the response to the incident? The recreation of the deployment uses our default settings of the HPA with a minimum of 2 replicas.  It took roughly 3 minutes until the container registry scaled up to it's original state prior to the incident.  At this point end users may have seen performance degradation.

## Timeline

2019-09-12

- 17:11:51 UTC - `kubectl delete gitlab-registry` - was executed on the production cluster
- 17:17:XX UTC - alerts indicating our registry endpoint alerted the on-call Engineer
- 17:19:XX UTC - Incident Call Started
- 17:23:59 UTC - Engineer recreated the Container Registry deployment
- 17:27:XX UTC - Alert clears

## Root Cause Analysis

An Engineer whom was performing local testing on an item directly related to the naming schema associated with objects that would match that of the production cluster.  Due to the nature of testing this removed a bit of context that would have immediately signaled to the Engineer that commands may have been sent to an undesired Kubernetes Cluster.  The engineer performed the command `kubectl delete deploy gitlab-registry` which matches the name of the deployment on the production cluster.  This deletion removes both the Deployment along with all Replicasets and Pods associated with it.  The Service Object remained, but no longer had any running Pods in its place.  HAProxy's healthcheck will now have signaled a failure and removed the `gke-registry` backend which left HAProxy to return HTTP 503's for all incoming requests.

## What went well

Start with the following:

- We were able to diagnose the issue relatively quickly
- With the use of Kubernetes, we were able to utilize our own tooling to bring the Container Registry to a working state quickly

## What can be improved

Start with the following:

- Using the root cause analysis, explain what can be improved to prevent this from happening again.
  * Engineers should not have unnecessary access to production clusters
- Is there anything that could have been done to improve the detection or time to detection?
  * Clusters should not have the API accessible by anyone other than service accounts acting inside of CI/CD jobs
  * Clusters should not allow API traffic from users from anywhere
  * A warning that an Engineer was acting upon a production cluster would caused a second guess at the commands being run locally
  * An indicator on the shell of the engineer may have warned the Engineer to which cluster they are connected too
- Is there anything that could have been done to improve the response or time to response?
  * Consider shortening the alert time required to trigger
  * An alert dedicated to the monitoring of the Pods/Replicaset/Deployment and whether it exists would've helped diagnose faster
- Is there an existing issue that would have either prevented this incident or reduced the impact?
  * No
- Did we have any indication or beforehand knowledge that this incident might take place?
  * No


## Corrective actions

An Epic has been created to discuss various ways of limiting exposure to production services in order to prevent future accidents such as this from occurring: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/91


## Guidelines

* [Blameless Postmortems Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 3.0