RCA: Note Creation on commit via API Calls Halted `new_note` Sidekiq Queue

<!--
**Please note:** if the incident relates to sensitive data, or is security related consider
labeling this issue with ~security and mark it confidential.
*** 
-->

Incident: gitlab-com/gl-infra/production#1028

## Summary

A user generated many notes on a single commit via API calls which slowed down the `new_note` sidekiq queue and blocked sending notifications for issue and MR comments for all customers.

- Service(s) affected : ~"Service:Sidekiq" 
- Team attribution : 
- Minutes downtime or degradation : 220

For calculating duration of event, use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/ZUei7TkWz/platform-metrics?orgId=1) to look at appdex and SLO violations.

## Impact & Metrics

Start with the following:

- What was the impact of the incident?
  - blocking to send out pending issue- and MR comment notifications
- Who was impacted by this incident?
  - all users supposed to get notifications
- How did the incident impact customers?
  - notifications arrived with a delay of up to 220m
- How many attempts were made to access the impacted service/feature?
- How many customers were affected?
- How many customers tried to access the impacted service/feature?

Include any additional metrics that are of relevance.

 - Up to 14,000 notifications queued up

Provide any relevant graphs that could help understand the impact of the incident and its dynamics.

![image](/uploads/7127adb9cf0f94cac69c30480b2fa1c1/image.png)

## Detection & Response

Start with the following:

- How was the incident detected?
  - [pagerduty alert](https://gitlab.pagerduty.com/incidents/PPF0U8C) for `new_note` queue size 
- Did alarming work as expected?
  - Alarming for queue size worked, but the alert threshold was reached over an hour after the incident started
- How long did it take from the start of the incident to its detection?
  - 68m
- How long did it take from detection to remediation?
  - 152m
- Were there any issues with the response to the incident?
  - it wasn't easy to get hold of a backend engineer at the beginning

## Root Cause Analysis

`new_note` notifications have been delayed.

1. Why? - They piled up in the `new_note` sidekiq queue. 
2. Why? - The queue was processing jobs considerably slower.
3. Why? - Many long-running jobs have been added to the queue.
4. Why? - A commit with a huge amount of comments got thousands of new comments and processing a commit with many comments is very slow.
5. Why? - Apparently an automation of a user was creating those comments via API inadvertently.

## What went well

Our monitoring detected the issue and paged the EOC. Backend engineers did a great job of finding and mitigating the root cause. Thanks to @andrew for jumping on the incident call quickly.

## What can be improved

- We should get alerted sooner when notes are stuck in the queue. E.g. not only alerting by queue size but also by job duration.
- Limit the amount of notes a user can create per time and on one commit.
- Make `new_note` processing more efficient on commits with many notes.
- We had the same kind of incident [a year ago](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/4230) and should have improved alerting and protections against too many notes since then.


## Corrective actions

* [ ] https://gitlab.com/gitlab-org/gitlab-ce/issues/46676
* [x] https://gitlab.com/gitlab-org/gitlab-ce/issues/60857
* [ ] improve alerting https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7752

## Guidelines

* [Blameless RCA Guideline](https://about.gitlab.com/handbook/infrastructure/#rca)
* [5 whys](https://en.wikipedia.org/wiki/5_Whys)

Story points: 2.0