[Spike] Error budget discrepancy against log data

#### Process:

1. Write the spike summary and requirement and select Definition of Done (DoD) questions that apply
1. DRI/s are assigned to the spike
2. Before starting the spike, engineer should confirm:
- [ ] Is the problem statement and description of the problem clear
- [ ] Are the requirements of this spike clear
- [ ] Are the Definition of done questions relevant/reasonable for this spike
3. Engineer does an investigation to gather information and note any initial impressions and possible paths for implementation
4. PM/EM/Collaborators + Engineer asks questions and start to narrow the scope to an appropriate solution/implementation
5. DoD questions are answered
6. DRI confirms next steps and the spike issue gets closed

------------------------

# Summary

Spike DRI/s: `@tanuki`

The Provision Error Budget panel seems to not reflect the request health rate computed from the logs. The following example is taken during Seat Link peak period, from 2023/1/19 03:00 UTC to 2023/1/19 05:00 UTC:

At that period, SeatLink requests is only over the latency threshold [46](https://log.gprd.gitlab.net/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2023-01-18T03:00:00.000Z',to:'2023-01-18T05:00:00.000Z'))&_a=(columns:!(),filters:!(),index:'7dbddf20-5530-11ed-b0ec-930003e0679c',interval:auto,query:(language:kuery,query:'json.controller%20:%20%22Api::V1::SeatLinksController%22%20and%20json.duration%20%3E%208500'),sort:!(!(json.time,desc)))) out of [5907](https://log.gprd.gitlab.net/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:'2023-01-18T03:00:00.000Z',to:'2023-01-18T05:00:00.000Z'))&_a=(columns:!(),filters:!(),index:'7dbddf20-5530-11ed-b0ec-930003e0679c',interval:auto,query:(language:kuery,query:'json.controller%20:%20%22Api::V1::SeatLinksController%22%20'),sort:!(!(json.time,desc)))) times, which should equal to 99% success apdex rate.

![Screenshot_2023-01-19_at_12-01-36_Discover_-_Elastic](/uploads/3bc9c835f13df2bc710afb3dd11c6e95/Screenshot_2023-01-19_at_12-01-36_Discover_-_Elastic.png)
![Screenshot_2023-01-19_at_12-02-01_Discover_-_Elastic](/uploads/7d0d7c80ce4c0cc5b3fa596492b829c1/Screenshot_2023-01-19_at_12-02-01_Discover_-_Elastic.png)

However Grafana Error Budget panel at this period seems to indicate otherwise. The [availability shown is down to 57.28%](https://dashboards.gitlab.net/d/stage-groups-detail-provision/stage-groups-provision-group-error-budget-detail?orgId=1&from=1674010800000&to=1674018000000).

![Screenshot_2023-01-19_at_11-59-58_stage-groups_Provision_group_error_budget_detail_-_Stage_Groups_-_Dashboards_-_Grafana](/uploads/a1e55ac2b277ea5e50018d39c163ab68/Screenshot_2023-01-19_at_11-59-58_stage-groups_Provision_group_error_budget_detail_-_Stage_Groups_-_Dashboards_-_Grafana.png)

The [apdex attribution chart](https://dashboards.gitlab.net/d/stage-groups-detail-provision/stage-groups-provision-group-error-budget-detail?orgId=1&from=1674010800000&to=1674018000000&viewPanel=44) also shows SeatLink as the major contributor and its unsuccessful apdex rate is down to -90%.

![Screenshot_2023-01-19_at_12-01-03_View_panel_-_stage-groups_Provision_group_error_budget_detail_-_Stage_Groups_-_Dashboards_-_Grafana](/uploads/bd855061020c767dfd0d529668002b9b/Screenshot_2023-01-19_at_12-01-03_View_panel_-_stage-groups_Provision_group_error_budget_detail_-_Stage_Groups_-_Dashboards_-_Grafana.png)

There seems to be a discrepancy between the log data and data in error budget panel.

Some discussion on the cause of this: https://gitlab.com/gitlab-org/customers-gitlab-com/-/issues/4884#note_1235160195

# Requirements / Ask

To investigate whether discrepancy does exist, and determine its root cause.

This may be easier for those with experience with Grafana/Prometheus queries, or metrics collection.

# Definition of done

What questions need to be answered in order for this spike to be completed - tick all those that apply:

- [ ] Is the ask possible?
- [ ] Is the work reasonable within time constraints?
- [x] What is the level of effort (t-shirt size estimate: S,M,L)?
- [ ] What are the risks / trade-offs?
- [ ] Where does complexity exist?
- [x] Are there any unknowns to be mindful of?
- [x] Is there a skills gap? Are there any documented reports of using this approach?
- [x] Is there a monitoring implication?
- [ ] Is there an availability implication?
- [ ] Is there a productivity implication?

# Next steps

Select the next steps that apply:

1. [ ] Close issue without further action = Spike conclusion is that feature won't be built
1. [ ] Create follow up issue to start MVC iteration of feature
1. [ ] Create MR for documentation update as a result of spike

Story points: 5.0