[RCA] Primary DB failover

## Summary

Postgres Primary restart or failover (https://gitlab.com/gitlab-com/gl-infra/production/issues/690).

Service(s) affected : ~"Service:Postgres"

Team attribution : Infrastructure

Minutes downtime or degradation : 2

## Impact & Metrics

- What was the impact of the incident?
  - increased error rates during failover, maybe slightly decreased db performance after incident because of missing table stats
- Who was impacted by this incident?
  - external customers
- How did the incident impact customers?
  - some customers might have seen error responses during failover
- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

GitLab Triage dashboard during incident:
https://dashboards.gitlab.net/d/RZmbBr7mk/gitlab-triage?orgId=1&from=1550091600000&to=1550092800000


## Detection & Response

Start with the following:

- How was the incident detected?
  - We got a short `IncreasedErrorRates` alert which wasn't associated with any DB problems. The incident then was mistaken as just missing table stats when we started to get "TooManyDeadTuples" alerts. A closer look into the DB logs on next day revealed that the primary DB did restart or failover.
- Did alarming work as expected?
  - We got alerted for higher error rates and dead tuples, so we knew that something was going on but we didn't get a notification for a DB failover and thus missed detecting the real cause until investigation on next day. 
- How long did it take from the start of the incident to its detection?
  - We detected higher error rates immediately. Detecting the cause took 14h.
- How long did it take from detection to remediation?
  - The DB cluster autonomously was up again within 2 minutes.
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team member wasn't page-able, ...)

## Timeline

2019-02-13

- 21:05 UTC - patroni04 fails to connect to consul
- 21:05 UTC - Postgres restarts on the patroni04
- 21:06 UTC - Increased Error Rates Alert (https://gitlab.pagerduty.com/incidents/PO854MA)
- 21:11 UTC - Increased Error Rates Alert resolves
- 21:48 UTC - Alert received `PostgreSQL_TooManyDeadTuples` on `ci_builds`
- 22:28 UTC - alert received `PostgreSQL_TooManyDeadTuples` on `ci_job_artifacts`
- 22:53 UTC - alert received `PostgreSQL_TooManyDeadTuples` on `ci_pipelines`

2019-02-14

- 01:33 UTC - alert received `PostgreSQL_TooManyDeadTuples` on `ci_stages`
- (more similar alerts)
- 10:24 UTC - noticed postgres restart has taken place at around 21.06pm UTC yesterday (spike in the memory graphs :trolleybus:) @abrandl
- 10:45 UTC - killed all ongoing `VACUUM ANALYZE` processes for two tables @abrandl
- 10:47 UTC - started `ANALYZE VERBOSE` on the full database, see https://gitlab.com/gitlab-com/gl-infra/production/issues/690#note_141280031 @abrandl
- 11:25 UTC - `ANALYZE VERBOSE` finished, statistics are back @abrandl
- 11:30 UTC - While our own investigation is going on, I asked OnGres to look as well (https://gitlab.slack.com/archives/CBCRJDSBY/p1550143723055000) @abrandl


## Root Cause Analysis

- Patroni demoted the primary postgres DB.
- Because it couldn't connect to the DCS (Consul).
- Because it got exceptions:
  - `HTTPConnectionPool(host='127.0.0.1', port=8500): Max retries exceeded with url: /v1/kv/service/pg-ha-cluster/?recurse=1`
  - `2019-02-13_21:05:23 patroni-04-db-gprd patroni[2192]:  2019-02-13 21:05:23,548 ERROR: Error communicating with DCS`
  - `2019-02-13 21:05:47,877 INFO: demoted self because DCS is not accessible and i was a leader`
  - `2019-02-13 21:05:50,524 INFO: promoted self to leader by acquiring session lock`
- Because patroni04 had network issues (maybe related to dmesg log showing TCP related kernel stack traces)

## What went well

- patroni failover worked very well within seconds
- we got alerts for higher error rates and dead tuples
- @yguo and @cshobe worked together to restore table stats
- @abrandl noticed the demotion

## What can be improved

- alerting for patroni demotion events
- missing table stats should lead to the conclusion that a db failover could have happened
- db ANALYZE procedures in case of DB failover should be better known now or be automated (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5841)
- monitoring kernel stack dumps might help to detect issues with networking

## Corrective actions

* [ ] Automatically run `ANALYZE` on failover (https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5841)
  - assignee: TBD
  - due: TBD
* [ ] Make sure we get notifications for Patroni restart/failover (even if the role does not change like here!) (#6232)
  - assignee: TBD
  - due: TBD
* [ ] Investigate cause of primary demotion (#6233)
  - assignee: TBD
  - due: TBD
* [ ] Consider monitoring of kernel stack trace messages in dmesg log
  - assignee: TBD
  - due: TBD

Story points: 2.0