Adjust SLO for `urgent-authorized-projects` sidekiq queue

## Summary

<!--
Give context for what problem this issue is trying to prevent from happening again.

Provide a brief assessment of the risk (chance and impact) of the problem that this corrective action fixes, to assist with triage and prioritization.
-->

Newly common alert: [`The shard_urgent_authorized_projects SLI of the sidekiq service (main stage) has an apdex violating SLO`](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7491)

Because of: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15955#note_1010193110

Requires: SLO adjustment

### Should this be moved to `throttled`

Looking at our [slos](https://gitlab.com/gitlab-com/runbooks/-/blob/c1f88c0a606e61aab56006f49359eaef6fe154e3/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L47-61) we have one called `throttled` where it doesn't look at `queue` time but only at `execution` time. 

The `urgent-authorized-projects` is set to `urgent=high` so it also looks at queue time, which is affecting our apdex.


#### Execution time

Looking at 1 day it seems like most jobs complete under 2 seconds.

![Screenshot_2022-08-02_at_09.36.32](/uploads/b5aea12f775b268af10c357ff2ad8a32/Screenshot_2022-08-02_at_09.36.32.png)

[Source](https://thanos.gitlab.net/graph?g0.expr=sum(sidekiq_jobs_execution_time%3A1m%7Benvironment%3D%22gprd%22%2C%20shard%3D~%22urgent-authorized-projects%22%7D)%20by%20(shard)%0A&g0.tab=0&g0.stacked=0&g0.range_input=1d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D)

Zooming out for 1 week we see the worst case scenario is `70s`

![Screenshot_2022-08-02_at_09.41.51](/uploads/8607f81b3522baf28cefab886b6ec392/Screenshot_2022-08-02_at_09.41.51.png)

[Source](https://thanos.gitlab.net/graph?g0.expr=sum(sidekiq_jobs_execution_time%3A1m%7Benvironment%3D%22gprd%22%2C%20shard%3D~%22urgent-authorized-projects%22%7D)%20by%20(shard)&g0.tab=0&g0.stacked=0&g0.range_input=1w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D)

#### Queue time

Looking at the queue time the p95 is around 10s, which is right around the [`urgent=high` slo](https://gitlab.com/gitlab-com/runbooks/-/blob/c1f88c0a606e61aab56006f49359eaef6fe154e3/metrics-catalog/services/lib/sidekiq-helpers.libsonnet#L49)

![Screenshot_2022-08-02_at_09.45.07](/uploads/605135208f1f992ca74b929f053c41e8/Screenshot_2022-08-02_at_09.45.07.png)

[Source](https://thanos.gitlab.net/graph?g0.expr=histogram_quantile(0.95%2C%20sum(sli_aggregations%3Asidekiq_jobs_queue_duration_seconds_bucket_rate5m%7Benvironment%3D%22gprd%22%2C%20shard%3D~%22urgent-authorized-projects%22%7D)%20by%20(le%2C%20shard))&g0.tab=0&g0.stacked=0&g0.range_input=1d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D)

Zooming out for 1 week we sometimes see it got to 60s

![Screenshot_2022-08-02_at_09.56.43](/uploads/12a1c80a24e85669f19e0dc917f56954/Screenshot_2022-08-02_at_09.56.43.png)

[Source](https://thanos.gitlab.net/graph?g0.expr=histogram_quantile(0.95%2C%20sum(sli_aggregations%3Asidekiq_jobs_queue_duration_seconds_bucket_rate5m%7Benvironment%3D%22gprd%22%2C%20shard%3D~%22urgent-authorized-projects%22%7D)%20by%20(le%2C%20shard))&g0.tab=0&g0.stacked=0&g0.range_input=1w&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D)

### Conclusion

The `urgent-authorized-projects` was created to throttle the number of jobs we execute concurrently from saturating the DB. It's unfortunate that we added `urgent` in the shard name because it's not `urgent` but it's throttled.



 

## Related Incident(s)

<!--
Note the originating incident(s) and link known related incidents/other issues.

The relation will happen automatically if you are creating this issue from an incident, if this isn't done already please uncomment the following line:
-->

Originating issue(s): gitlab-com/gl-infra/production#7491


## Desired Outcome/Acceptance Criteria

<!--
How will you know that this issue is complete?

If you have any initial thoughts on implementation details (e.g. what to do or not do, gotchas, edge cases etc.), please share them while they are fresh in your mind.
-->

This `urgent` classified activity is no longer completed within the expectations for `urgent` work, because it is being deliberately throttled with a lower concurrency level and less replicas.  It is not desired to reclassify this workload, but instead simply adjust the SLO for this particular queue so that it no longer alerts EOCs.

What we should do is:

- [x] Change the queue to `throttled` inside of the runbook :point_right: https://gitlab.com/gitlab-com/runbooks/-/merge_requests/4860
- [x] Update the label from `urgent=high` to `urgent=throttled` so the `throttled` SLO is taken into consideration.
    -  [x] `gstg`: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/942
    -  [x] `gprd`: https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-helmfiles/-/merge_requests/944
- [x] Create follow up issue to clean this up with the shard itself (since the shard will be deleted once we fix https://gitlab.com/groups/gitlab-org/-/epics/8200) :point_right: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/16080#note_1065636788

## Associated Services

<!--
Apply the appropriate services associated with this corrective action if applicable.

~Service::SERVICE_NAME

-->

## Corrective Action Issue Checklist

* [x] Link the incident(s) this corrective action arose out of
* [x] Give context for what problem this corrective action is trying to prevent from re-occurring
* [x] Assign a severity label (this is the highest sev of related incidents, defaults to 'severity::4')
* [x] Assign a priority (this will default to 'priority::4')

Story points: 1.0