Discuss: How to make SLO alerts for CI Runners less noisy and more actionable?

The SLO alerts for CI Runners does not yet meet reasonable standards for an actionable alert.  We all know this, but I wanted to jot down a few specific things that would make it better.

- This alert still has an unreasonably low signal-to-noise ratio.
    - The apdex score dropped below SLO [9 times in the last 24 hours](https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1&from=1571420700000&to=1571507100000&var-PROMETHEUS_DS=Global&var-environment=gprd&var-type=ci-runners&var-stage=main&var-sigma=2&fullscreen&panelId=7), 5 of which lasted long enough for PagerDuty to alert (see [screenshot as of 17:45 UTC](/uploads/4bb87a9e4b688fe32461617bc310bcd1/Screenshot_from_2019-10-19_10-56-08.png)).
- This alert has no runbook yet.
    - The PagerDuty alert refers to `troubleshooting/service-ci-runners.md`, but that file does not exist in the [runbooks repo](https://ops.gitlab.net/gitlab-com/runbooks/tree/master/troubleshooting).
    - The PagerDuty alert itself gives a recommended response which amounts to: look at other metrics and logs for signs of abusive behavior, high request rate, or slow responses from dependent services.  That's good generic advice for any SLO violation, but it's too vague to be actionable.
- What dashboards are useful for troubleshooting CI Runners?
    - The ["CI Runners Service" -> "CI" dashboard](https://dashboards.gitlab.net/d/000000159/ci?orgId=1&refresh=5m) shows workload-related metrics:
        - queue sizes
        - percentiles of durations of recently completed jobs
        - job count by runner node
    - The ["Service Platform Metrics" dashboard](https://dashboards.gitlab.net/d/general-service/general-service-platform-metrics?orgId=1&var-PROMETHEUS_DS=Global&var-environment=gprd&var-type=ci-runners&var-stage=main&var-sigma=2&fullscreen&panelId=7) can show the latency apdex graph when `type` is set to `ci-runners`.
- Should the SLO should be reframed in a way that makes it more achievable?
    - The following are just my initial thoughts, but I believe @andrewn is already advocating for improving this, and he probably already has much more specific ideas for improvements.
    - The CI Runner workload is largely outside of our control (e.g. user-defined jobs may take any amount of time to complete, and they may depend on external resources that are outside of both our control and visibility.  Example: Would SLO be violated by a sufficiently large number of jobs running `sleep 1000`?
    - Reframing the SLO in terms that address *only the controllable aspects* of our platform's execution of jobs would make this goal more achievable.  What working agreements do we already have for the CI Runner platform, describing to our end-users under what conditions a job execution may be aborted, such as job timeout deadline, usage quotas on machine resources (CPU, network bandwidth, max connection count, max disk IOPS, etc.)?
    - We may benefit from comparing our set of quotas to those of other platform providers (e.g. GCP, AWS, Heroku, etc.).  Other platform providers also share this problem of needing to set achievable SLOs that are resilient to abusive behavior and support auto-scaling, auto-termination, and auto-rate-limiting.
- Both auto-scaling and building automatic defensive mitigations for abuse requires having more specific goals than just a latency target for an unconstrained input workload.  More prescriptively describing the service level would also be super helpful in making concrete proposals for automating tactics to achieve these goals while also improving cost efficiency and reducing toil.

cc @gitlab-com/gl-infra

Story points: 1.0