Readiness Review for Jaeger

This checklist is sourced from https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/.gitlab/issue_templates/production_readiness.md.

Link to the Jaeger runbook, which covers majority of the points listed below: https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/jaeger/README.md

The scope of this review is:
- deploying Jaeger to the production environment
- Elasticsearch instance deployed to Elastic Cloud (used as storage backend for the Jaeger service)
- Labkit instrumentation

## Summary

* [x] Short overview mentioning purpose of the service, dependencies and owners
* [x] Explain the scope of this review and what is explicitly out of scope.

## Architecture

* [x] Runbook README.md contains an architecture overview (provide link)
  * [x] Runbook README.md contains a logical architecture diagram
  * [x] Runbook README.md contains a physical architecture diagram (optional)
  * [x] Runbook README.md provides enough information for a reviewer to get an
    understanding of the service and it's components, dependencies and
    interactions

## Documentation

* [x] is there a blue print/design doc? (provide link)
* [x] do we have runbooks? (provide links)
* [x] are runbooks up-to-date?
* [x] where else is documentation for this service located?
* [ ] is there a service catalog entry? (provide link)
  * [x] is service catalog listing all dependencies?
  * [ ] has service catalog links to all existing documentation?
  * [ ] is service catalog linking to readiness review?

## Performance

* [x] is there a runbook section with performance characteristics? (it should
  cover following considerations, provide link)
  * [x] current requests/s (min, max, average), latency characteristics,
    saturation, ...
  * [x] throtteling/limits
  * [x] bottlenecks (cpu-bound, memory-bound, ...)
  * [x] is there documentation on how/why we set certain config options that are
    affecting performance?

## Scalability

* [x] is there a runbook section with scalability information? (it should cover
  following considerations, provide link)
  * [x] expected load in the future
  * [x] how can we scale to the expected load?
  * [x] can it be scaled across availability zones or regions?
  * [x] are there scalability limitations?
  * [x] are we doing performance tests?

## Availability

* [x] is there a runbook section covering availability considerations? (it
  should cover following topics, provide link)
  * [x] failure modes of this service, blast radius, how long does it take to
    recover?
  * [x] what happens on outage of services we are depending on?
  * [x] Availability Zone (AZ) outage
  * [x] split brain between AZs
  * [x] region outage
  * [x] other external dependencies that could affect availability
  * [x] what other services are affected by an outage of this service?
  * [x] is there an existing Recovery Time Objective (RTO) documented? How do we
    plan to achieve it?
  * [x] do we have an error budget?
  * [x] are we doing disaster recovery tests?
  * [x] is there a failover procedure? Do we have runbook instructions?

## Durability

* [x] is there a runbook section covering durability considerations? (it should
  cover following topics, provide link)
* [x] possible failure modes and how to recover from them
  * [x] deletion by accident
  * [x] disk failure
  * [x] data corruption
  * [x] GCP outage
  * ...
  * [x] is there an existing Recovery Point Objective (RPO) documented? How do
    we plan to achieve it?
  * Backups
    * [x] are we testing backup replay?
    * [x] are we monitoring backups?
    * [x] what is the backup retention policy?
    * [x] are backups in a different logical and physical environment?

## Security/Compliance

* [ ] is there a runbook section covering security considerations? (it should
  cover following topics, provide link)
  * [x] list of access roles
  * [x] Who has which role?
  * [x] How do we protect access?
  * [x] Auditability of access
  * [x] Which entrypoints need protection?
  * [x] How are we applying security updates? (OS and service)
  * [ ] Regulations/Policies applying? (PII, SOX, ...)
  * [x] how do we protect customer data?
    * [x] encryption at rest?
    * [x] could customer data leak in logs?
    * [x] how long do we keep logs?
* [ ] is someone from security included for the readiness review?

## Monitoring

* [x] is there a runbook section covering monitoring? (it should
  cover following topics, provide link)
  * [x] list key SLIs. Are we monitoring them?
  * [x] list SLOs. Are we monitoring/alerting on them?
  * [x] list of relevant alerts
  * [x] are alerts actionable and linking to a runbook?
  * [x] do we have a metrics catalog entry for the service? (provide link)
  * [x] list of relevant dashboards
  * [x] list of relevant logs

Story points: 5.0