[RCA] Loss of db metrics visibility by switching to dedicated prometheus instance for db metrics

## Summary

By starting to scrape db metrics from a new dedicated prometheus instance our grafana db dashboards stopped working.

Service(s) affected : ~"Service:Postgres"

Team attribution : Infrastructure

Minutes downtime or degradation : 21h

## Impact & Metrics

Start with the following:

- What was the impact of the incident?
  - We lost visibility into db metrics for a while (although metrics were still scraped, but by another instance, which wasn't noticed)
- Who was impacted by this incident?
  - SRE oncall, release team - having to delay the 11.8 release by an hour.
- How did the incident impact customers?
  - no customer impact

## Detection & Response

Start with the following:

- How was the incident detected?
  - we got alerts for missing metrics
- Did alarming work as expected?
  - we got some alerts for missing metrics data, but not specific enough to immediately be aware of what was going on.
- How long did it take from the start of the incident to its detection?
  - 12h
- How long did it take from detection to remediation?
  - 9h
- Were there any issues with the response to the incident? (i.e. bastion host used to access the service was not available, relevant team memeber wasn't page-able, ...)
  - we should have noticed earlier from the alerts that db monitoring wasn't working anymore
  - paging for DBRE support didn't work as DBRE schedules were not communicated clear enough and escalation was still going to OnGres (which doesn't support us anymore)

## Timeline

2019-02-21

- 17:35 UTC - general-alerts: `Operation rate data for the "service" component of the "patroni" service is missing`

2019-02-22

- 05:29 UTC - @andrewn raising concern about above alert
- 12:34 UTC - incident issue created (https://gitlab.com/gitlab-com/gl-infra/production/issues/697)
- 12:58 UTC - DBRE oncall was paged, team starts fixing Grafana dashboards
- 13:24 UTC - release team wants to deploy 11.8, SRE oncall is holding them off to first get db metrics visibility again
- 14:29 UTC - SRE oncall gives green light for 11.8 deploy


## Root Cause Analysis

general db alerts didn't get data anymore.
Because db dashboards in grafana didn't work anymore.
Because prometheus-gprd wasn't scraping db data anymore.
Because a change was made to scrape them from a new, dedicated instance, prometheus-db.
Because db metrics were making up most of the load on prometheus-gprd.

## What went well

- general-alerts warning for missing db metrics data
- @andrewn noticing the alerts and taking action, making team aware of the consequences and creating an incident
- @ahmadsherif and @yguo jumping in to adjust the dashboards

## What can be improved

- Better communication to make everybody aware of changes in monitoring
- Making sure that dashboards and alerts are still working when changing prometheus setup
- Better response to alerts that are not fully understood (as the missing metrics alert)

## Corrective actions

- Fix Grafana Dashboards https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6266
- Check if we still get all the db metrics and alerts via prometheus-db: https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6271
- Communicate the new prometheus-db instance

Story points: 2.0