[Incident Review] 2020-04-22 Lack of Observability

Incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1978


## Summary

Prometheus and alertmanager were still operational, Thanos was completely unavailable and because of that also Grafana

- no alerts were triggered:
  - we could use Elastic watcher [http input](https://www.elastic.co/guide/en/elasticsearch/reference/current/input-http.html) to perform a simple check if Prometheus, Alertmanager, Thanos, Grafana are operational. @bjk-gitlab do we already have any alerting for the "monitoring stack" external to it?
- what steps were taken to troubleshoot it? (how can we improve time to detection)
  - we have this dashboard for the monitoring components: https://dashboards.gitlab.net/d/monitoring-main/monitoring-overview?orgId=1&from=now-6h%2Fm&to=now%2Fm&var-PROMETHEUS_DS=Global&var-environment=gprd&var-stage=main&var-sigma=2 . However, a lot of the charts are empty. Besides, these should be monitored by an external system
  - We have Thanos sending tracing info to Elastic APM
  - The only logs available in Kibana are from Prometheus: https://log.gprd.gitlab.net/goto/db7eca8ec4cbd80b2ae1187715e62984
- what would have prevented it from happening?
  - the change was reviewed, we simply missed the fact that it resulted in a circular dependency

1. Service(s) affected : ~"Service::Thanos" 
1. Team attribution : ~"team::Observability" 
1. Minutes downtime or degradation : ~600

## Customer Impact

1. Who was impacted by this incident? **All employees attempting to use dashboards.gitlab.net and the infrastructure department's monitoring systems.**
2. What was the customer experience during the incident? **None**
3. How many customers were affected? **None**
4. If a precise customer impact number is unknown, what is the estimated potential impact? **n/a**

## Incident Response Analysis

1. How was the event detected? **Grafana dashboards were timing out.**
2. How could detection time be improved? **Monitoring of the Grafana system or latency for Thanos queriies could have alerted us.**
3. How did we reach the point where we knew how to mitigate the impact?
4. How could time to mitigation be improved?


## Post Incident Analysis

1. How was the root cause diagnosed?
2. How could time to diagnosis be improved?
3. Do we have an existing backlog item that would've prevented or greatly reduced the impact of this incident? **Yes, https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/159.**
4. Was this incident triggered by a change (deployment of code or change to infrastructure. _if yes, have you linked the issue which represents the change?_)? **Yes, https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2135.** 
   


## Timeline

All times UTC.

2020-04-21
- 19:00:00 https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2135 change is made to rules which results in a circular dependency in Thanos
- 23:40:00 transaction durations start to go up

2020-04-22

- 02:15:00 request duration goes through the roof, we start to hit the 5min timeout
- 08:59:00 https://gitlab.com/gitlab-cookbooks/gitlab-prometheus/-/merge_requests/521 adds a monitor label to the rule server
- 09:02:00 https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2140/diffs  adds a filter to record rules to skip the rule server
- 10:29:00 https://gitlab.com/gitlab-com/runbooks/-/merge_requests/2142 improving routing


## 5 Whys
<!--
_This section is meant to dig into lessons learned and corrective actions, it is not limited to 5 and consider how you may dive deeper into each why_

_example:_

1. Customers experienced an inability to create new projects on GitLab.com, why?
   - A code change was deployed which contained an escaped bug.
1. Why did this bug not get noticed in staging?
   - The integration test for this use case is missing.
1. Why is an integration test for this use case missing?
   - It was inadvertently removed during a refactoring of our test suite.
1. Why was the test suite being refactored?
   - As part of our efforts to decrease MTTP.
1. Why did it take 2 hours to notice this issue in production?
   - The initial alert was supressed as a false alarm.
1. Why was this alert suppressed
   - The system which dedupes alerts inadvertently suppressed this alarm as a duplicate. 
1. Why did it take 4 hours to resolve the issue in production?
   - The change which carried this escaped bug also contained a database schema change which made rolling the change back impossible. Engineering was engaged immediately by the oncall SRE and conducted a forward fix. 
-->

## Lessons Learned
<!--
_Be explicit about what lessons we learned and should carry forward. These usually inform what our corrective actions should be._

_example:_
1. The results of refactoring activites around our integration tests should be reviewed. (i.e we had 619 tests before refactor but 618 after.)
2. Our tooling to dedupe alarms should have integration tests to ensure it works against existing and newly added alarms.  
-->

## Corrective Actions

- investigate and fix monitoring related metrics which are unavailable in Grafana
  - issue: https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10027
- add an Alertmanager alert for Thanos latency (might be a matter of defining SLO threshold)
  - there's already an epic: https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/159
- add Alertmnager alerts for other components of the monitoring stack
  - same as above
- add Elastic watches for alerting when the monitoring stack is down
  - same as above
- start using Jaeger/Elastic APM for tracing Prometheus/Alertmanager/Grafana
  - support for Jaeger in Prometheus is coming in next release
- send logs from Thanos, alertmanager, Grafana to Kibana
  - they simply don't log a lot
- set up a staging environment for our monitoring stack so that we can test changes before they go to production (I think that this would be an overkill, we should focus on better monitoring and alerting instead)
  - we should do it as part of the migration to Kubernetes, creation of the staging env will be easier when not done with Chef and will help the migration itself as well

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html#meeting-purpose)

Story points: 1.0