Repository: home-assistant/core
Pull request: Fix high latency from thundering heard at 0 microseconds
## Proposed change
<!--
  Describe the big picture of your changes here to communicate to the
  maintainers why we should accept this pull request. If it fixes a bug
  or resolves a feature request, be sure to link to that issue in the
  additional information section.
-->
Fix high latency at 0 microseconds
fixes #82231

While [testing bluetooth connect times](https://ptb.discord.com/channels/330944238910963714/1012073425638015037/1042549387374641182), I noticed the latency goes way up at 0 microseconds.  All the slower connect times were around the 0 microseconds mark.  It became clear that we have a thundering heard problem with our scheduling. 

To solve this I used a strategy similar to the one outlined in rfc6762 [5.2](https://www.rfc-editor.org/rfc/rfc6762.html#section-5.2). to avoid accidental synchronization.  We now choose a random microsecond between 50000 and 500000 for repeating listeners that don't specify a microsecond.   This ensures they are still likely to finish within the second boundary (unless they take a long time and than it probably doesn't matter anyways) and updates are now staggered to avoid them all happening at the 0 microsecond.

The effect of avoiding the thundering heard was even visible on my system which is mostly local push and doesn't have much polling.  I expect the improvement will smooth out some momentary stalling and contention on systems that have significant polling.

After this change bluetooth connect times are more consistent regardless of which microsecond the connection request gets started at.

![IMG_0070](https://user-images.githubusercontent.com/663432/202468786-d41ce058-13a0-42f4-8832-0b463c775bb9.png)


## Type of change
<!--
  What type of change does your PR introduce to Home Assistant?
  NOTE: Please, check only 1! box!
  If your PR requires multiple boxes to be checked, you'll most likely need to
  split it into multiple PRs. This makes things easier and faster to code review.
-->

- [ ] Dependency upgrade
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New integration (thank you!)
- [ ] New feature (which adds functionality to an existing integration)
- [ ] Deprecation (breaking change to happen in the future)
- [ ] Breaking change (fix/feature causing existing functionality to break)
- [x] Code quality improvements to existing code or addition of tests

## Additional information
<!--
  Details are important, and help maintainers processing your PR.
  Please be sure to fill out additional details, if applicable.
-->

- This PR fixes or closes issue: fixes #
- This PR is related to issue: 
- Link to documentation pull request: 

## Checklist
<!--
  Put an `x` in the boxes that apply. You can also fill these out after
  creating the PR. If you're unsure about any of them, don't hesitate to ask.
  We're here to help! This is simply a reminder of what we are going to look
  for before merging your code.
-->

- [x] The code change is tested and works locally.
- [ ] Local tests pass. **Your PR cannot be merged unless tests pass**
- [ ] There is no commented out code in this PR.
- [ ] I have followed the [development checklist][dev-checklist]
- [ ] The code has been formatted using Black (`black --fast homeassistant tests`)
- [x] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

- [ ] Documentation added/updated for [www.home-assistant.io][docs-repository]

If the code communicates with devices, web services, or third-party tools:

- [ ] The [manifest file][manifest-docs] has all fields filled out correctly.  
      Updated and included derived files by running: `python3 -m script.hassfest`.
- [ ] New or updated dependencies have been added to `requirements_all.txt`.  
      Updated by running `python3 -m script.gen_requirements_all`.
- [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.
- [ ] Untested files have been added to `.coveragerc`.

<!--
  This project is very active and we have a high turnover of pull requests.

  Unfortunately, the number of incoming pull requests is higher than what our
  reviewers can review and merge so there is a long backlog of pull requests
  waiting for review. You can help here!
  
  By reviewing another pull request, you will help raise the code quality of
  that pull request and the final review will be faster. This way the general
  pace of pull request reviews will go up and your wait time will go down.
  
  When picking a pull request to review, try to choose one that hasn't yet
  been reviewed.

  Thanks for helping out!
-->

To help with the load of incoming pull requests:

- [ ] I have reviewed two other [open pull requests][prs] in this repository.

[prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure

<!--
  Thank you for contributing <3

  Below, some useful links you could explore:
-->
[dev-checklist]: https://developers.home-assistant.io/docs/en/development_checklist.html
[manifest-docs]: https://developers.home-assistant.io/docs/en/creating_integration_manifest.html
[quality-scale]: https://developers.home-assistant.io/docs/en/next/integration_quality_scale_index.html
[docs-repository]: https://github.com/home-assistant/home-assistant.io


Problem statement:
Event loop has higher latency at 0 microsecond timestamps
### The problem

The latency goes way up at 0 microseconds.  It seems we have some type of thundering herd problem where many events align to happen at 0

### What version of Home Assistant Core has the issue?

dev

### What was the last working version of Home Assistant Core?

n/a

### What type of installation are you running?

Home Assistant Core

### Integration causing the issue

_No response_

### Link to integration documentation on our website

_No response_

### Diagnostics information

_No response_

### Example YAML snippet

_No response_

### Anything in the logs that might be useful for us?

_No response_

### Additional information

_No response_

Hints from issue discussion:
Well as soon as I opened the issue I found the issue

#32974 introduced code to align all update coordinators at microsecond 0

Commit message: Fix high latency at 0 microseconds

fixes #82231

Patch to review:
diff --git a/homeassistant/helpers/update_coordinator.py b/homeassistant/helpers/update_coordinator.py
index 768b8040729ecc..e344585cc8ccbd 100644
--- a/homeassistant/helpers/update_coordinator.py
+++ b/homeassistant/helpers/update_coordinator.py
@@ -5,6 +5,7 @@
 from collections.abc import Awaitable, Callable, Coroutine, Generator
 from datetime import datetime, timedelta
 import logging
+from random import randint
 from time import monotonic
 from typing import Any, Generic, TypeVar
 import urllib.error
@@ -61,6 +62,10 @@ def __init__(
         # when it was already checked during setup.
         self.data: _T = None  # type: ignore[assignment]
 
+        # Pick a random microsecond to stagger the refreshes
+        # and avoid a thundering herd.
+        self._microsecond = randint(50000, 500000)
+
         self._listeners: dict[CALLBACK_TYPE, tuple[CALLBACK_TYPE, object | None]] = {}
         self._job = HassJob(self._handle_refresh_interval)
         self._unsub_refresh: CALLBACK_TYPE | None = None
@@ -138,11 +143,17 @@ def _schedule_refresh(self) -> None:
         # We _floor_ utcnow to create a schedule on a rounded second,
         # minimizing the time between the point and the real activation.
         # That way we obtain a constant update frequency,
-        # as long as the update process takes less than a second
+        # as long as the update process takes less than 500ms
+        #
+        # We do not align everything to happen at microsecond 0
+        # since it increases the risk of a thundering herd
+        # when multiple coordinators are scheduled to update at the same time.
+        #
+        # https://github.com/home-assistant/core/issues/82231
         self._unsub_refresh = event.async_track_point_in_utc_time(
             self.hass,
             self._job,
-            utcnow().replace(microsecond=0) + self.update_interval,
+            utcnow().replace(microsecond=self._microsecond) + self.update_interval,
         )
 
     async def _handle_refresh_interval(self, _now: datetime) -> None:


Code review comments:
diff_hunk: @@ -61,6 +62,10 @@ def __init__(
         # when it was already checked during setup.
         self.data: _T = None  # type: ignore[assignment]
 
+        # Pick a random microsecond to stagger the refreshes
+        # and avoid a thundering herd.
+        self._microsecond = randint(50000, 500000)
path: homeassistant/helpers/update_coordinator.py
text: @author:
I picked a minimum of 50000 microseconds to avoid aligning things to 0 since there are other places this happens given `track_time_interval` also has this issue since `dt_util.find_next_time_expression_time` zeros out microseconds

@author:
I guess I should fix that as well

@author:
I just happen to have enough coordinators to get it to happen.  Someone else is going to have enough interval trackers