CD/D Readiness Review Part 4 - Performance, Monitoring and Alerting.

Part 4 of CI/CD to SRE.

## Performance

- [ ] Explain what validation was done following GitLab's [performance guidlines](https://docs.gitlab.com/ce/development/performance.html) please explain or link to the results below
    * [Query Performer](https://docs.gitlab.com/ce/development/query_recorder.html)
    * [Sherlock](https://docs.gitlab.com/ce/development/profiling.html#sherlock)
    * [Request Profiling](https://docs.gitlab.com/ce/administration/monitoring/performance/request_profiling.html)
- [ ] Are there any potential performance impacts on the database when this feature is enabled at GitLab.com scale?
- [ ] Are there any throttling limits imposed by this feature? If so how are they managed?
- [ ] If there are throttling limits, what is the customer experience of hitting a limit?
- [ ] For all dependencies external and internal to the application, are there retry and back-off strategies for them?
- [ ] Does the feature account for brief spikes in traffic, at least 2x above the expected TPS?


## Monitoring and Alerts

- [ ] Is the service logging in JSON format and are logs forwarded to logstash?
- [ ] Is the service reporting metrics to Prometheus?
- [ ] How is the end-to-end customer experience measured?
- [ ] Do we have a target SLA in place for this service?
- [ ] Do we know what the indicators (SLI) are that map to the target SLA?
- [ ] Do we have alerts that are triggered when the SLI's (and thus the SLA) are not met?
- [ ] Do we have troubleshooting runbooks linked to these alerts?
- [ ] What are the thresholds for tweeting or issuing an official customer notification for an outage related to this feature?

## Responsibility

- [ ] Which individuals are the subject matter experts and know the most about this feature?
- [ ] Which team or set of individuals will take responsibility for the reliability of the feature once it is in production?
- [ ] Is someone from the team who built the feature on call for the launch? If not, why not?


## Testing

- [ ] Describe the load test plan used for this feature. What breaking points were validated?
- [ ] For the component failures that were theorized for this feature, were they tested? If so include the results of these failure tests.
- [ ] Give a brief overview of what tests are run automatically in GitLab's CI/CD pipeline for this feature?

Existing docs to use as reference:
- Notes from recent summary meeting - https://docs.google.com/document/d/1diT_Dt8oE2kls09WZg6YU3twuQMM6u6NmseCsB-IlXg/edit
- Diagrams, etc - https://docs.google.com/document/d/1WYmN5oukY3DK2hPFLPkxwnuyfxES8nNPeDLMTN_KhVM/edit

Acceptance Criteria:
- [ ] Perform the monitoring and alerting analysis above.  Gather notes and link to the service catalog:  https://gitlab.com/gitlab-com/runbooks/blob/master/services/service-catalog.yml
Child issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6199
- [ ] In this case, make no changes to actual alerting infra, just prep up a Design MR for the new routing of alerts to be done once we have updated all of the SRE team with the proper information, links to runbooks, etc.

Child issue of https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/6199

Story points: 2.0