Repository: home-assistant/core
Pull request: Reduce the number of API calls in Twitch integration
I follow almost 200 people.  This reduces the number of API calls from around 800 to around 404.

<!--
  You are amazing! Thanks for contributing to our project!
  Please, DO NOT DELETE ANY TEXT from this template! (unless instructed).
-->
## Breaking change
<!--
  If your PR contains a breaking change for existing users, it is important
  to tell them what breaks, how to make it work again and why we did this.
  This piece of text is published with the release notes, so it helps if you
  write it towards our users, not us.
  Note: Remove this section if this PR is NOT a breaking change.
-->


## Proposed change
<!--
  Describe the big picture of your changes here to communicate to the
  maintainers why we should accept this pull request. If it fixes a bug
  or resolves a feature request, be sure to link to that issue in the
  additional information section.
-->


## Type of change
<!--
  What type of change does your PR introduce to Home Assistant?
  NOTE: Please, check only 1! box!
  If your PR requires multiple boxes to be checked, you'll most likely need to
  split it into multiple PRs. This makes things easier and faster to code review.
-->

- [ ] Dependency upgrade
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New integration (thank you!)
- [ ] New feature (which adds functionality to an existing integration)
- [ ] Deprecation (breaking change to happen in the future)
- [ ] Breaking change (fix/feature causing existing functionality to break)
- [x] Code quality improvements to existing code or addition of tests

## Additional information
<!--
  Details are important, and help maintainers processing your PR.
  Please be sure to fill out additional details, if applicable.
-->

- This PR fixes or closes issue: fixes #128866
- This PR is related to issue: 128866
- Link to documentation pull request: 

## Checklist
<!--
  Put an `x` in the boxes that apply. You can also fill these out after
  creating the PR. If you're unsure about any of them, don't hesitate to ask.
  We're here to help! This is simply a reminder of what we are going to look
  for before merging your code.
-->

- [x] The code change is tested and works locally.
- [ ] Local tests pass. **Your PR cannot be merged unless tests pass**
- [x] There is no commented out code in this PR.
- [x] I have followed the [development checklist][dev-checklist]
- [x] I have followed the [perfect PR recommendations][perfect-pr]
- [x] The code has been formatted using Ruff (`ruff format homeassistant tests`)
- [ ] Tests have been added to verify that the new code works.

If user exposed functionality or configuration variables are added/changed:

- [ ] Documentation added/updated for [www.home-assistant.io][docs-repository]

If the code communicates with devices, web services, or third-party tools:

- [ ] The [manifest file][manifest-docs] has all fields filled out correctly.  
      Updated and included derived files by running: `python3 -m script.hassfest`.
- [ ] New or updated dependencies have been added to `requirements_all.txt`.  
      Updated by running `python3 -m script.gen_requirements_all`.
- [ ] For the updated dependencies - a link to the changelog, or at minimum a diff between library versions is added to the PR description.

<!--
  This project is very active and we have a high turnover of pull requests.

  Unfortunately, the number of incoming pull requests is higher than what our
  reviewers can review and merge so there is a long backlog of pull requests
  waiting for review. You can help here!
  
  By reviewing another pull request, you will help raise the code quality of
  that pull request and the final review will be faster. This way the general
  pace of pull request reviews will go up and your wait time will go down.
  
  When picking a pull request to review, try to choose one that hasn't yet
  been reviewed.

  Thanks for helping out!
-->

To help with the load of incoming pull requests:

- [ ] I have reviewed two other [open pull requests][prs] in this repository.

[prs]: https://github.com/home-assistant/core/pulls?q=is%3Aopen+is%3Apr+-author%3A%40me+-draft%3Atrue+-label%3Awaiting-for-upstream+sort%3Acreated-desc+review%3Anone+-status%3Afailure

<!--
  Thank you for contributing <3

  Below, some useful links you could explore:
-->
[dev-checklist]: https://developers.home-assistant.io/docs/development_checklist/
[manifest-docs]: https://developers.home-assistant.io/docs/creating_integration_manifest/
[quality-scale]: https://developers.home-assistant.io/docs/integration_quality_scale_index/
[docs-repository]: https://github.com/home-assistant/home-assistant.io
[perfect-pr]: https://developers.home-assistant.io/docs/review-process/#creating-the-perfect-pr


Problem statement:
Twitch integration uses excessive API requests
### The problem

Many of the Helix API requests (e.g. `get-streams`, `get-channel-followers`, and `get-followed-channels`) offer pagination.  I follow about 120 channels, which ends up being somewhere in the area of 500 API requests and takes about 3 minutes to update every channel.

For me, the number of API requests could be reduced by 75% by adding pagination.  Further, if usage of `check-user-subscription` (no pagination available) became a config option, the number of API requests would be reduced to only about 10.

### What version of Home Assistant Core has the issue?

core-2024.10.3

### What type of installation are you running?

Home Assistant OS

### Integration causing the issue

twitch

Hints from issue discussion:
I'm wrong about `get_streams`; it would need to switch to `get_followed_streams` to get pagination (`get_streams` works on all streams on the platform and removing `broadcaster_id` there would just be a massive list).

I took a crack at this, but async Python breaks my brain and I gave up.  Taking `get_channel_followers` and `get_followed_streams` out of the `self.users` loop did seem to work though -- you just need to remove `channel.id` and change/remove the `first` param.
I mean I am happy to help you out where needed. I use Twitch not that often so it's not that high on my priority list (which is long), so if I can help you out with that, that would be awesome
[twitch documentation](https://www.home-assistant.io/integrations/twitch)
[twitch source](https://github.com/home-assistant/core/tree/dev/homeassistant/components/twitch)
I can open a half-baked PR this evening.  It won't be right, but it'll be
close.  It should reduce API calls by about 50% the way I did it.

On Mon, Oct 21, 2024, 2:43 PM home-assistant[bot] ***@***.***>
wrote:

> twitch documentation <https://www.home-assistant.io/integrations/twitch>
> twitch source
> <https://github.com/home-assistant/core/tree/dev/homeassistant/components/twitch>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/home-assistant/core/issues/128866#issuecomment-2427459209>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAFUDNOZTYSDZCJKF7FJVDDZ4VDNNAVCNFSM6AAAAABQJACQG2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRXGQ2TSMRQHE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>

Also be aware, something got merged that centralizes the api calls, so there already is a difference between 2024.10 and `dev`. It probably won't be fabulous, since we need to store all the data in 1 entity (which I want to split up, hence I centralized the data fetching)
PR #128996

Commit message: Make tests pass after Twitch API reduction changes.

Patch to review:
diff --git a/homeassistant/components/twitch/coordinator.py b/homeassistant/components/twitch/coordinator.py
index 5e3de4c4ec8ea..6390ca45e867d 100644
--- a/homeassistant/components/twitch/coordinator.py
+++ b/homeassistant/components/twitch/coordinator.py
@@ -4,7 +4,7 @@
 from datetime import datetime, timedelta
 
 from twitchAPI.helper import first
-from twitchAPI.object.api import FollowedChannelsResult, TwitchUser, UserSubscription
+from twitchAPI.object.api import FollowedChannel, Stream, TwitchUser, UserSubscription
 from twitchAPI.twitch import Twitch
 from twitchAPI.type import TwitchAPIException, TwitchResourceNotFound
 
@@ -83,11 +83,23 @@ async def _async_update_data(self) -> dict[str, TwitchUpdate]:
             False,
         )
         data = {}
+        streams: list[Stream] = [
+            s
+            async for s in self.twitch.get_followed_streams(
+                user_id=self.current_user.id, first=100
+            )
+        ]
+        follows: list[FollowedChannel] = [
+            f
+            async for f in await self.twitch.get_followed_channels(
+                user_id=self.current_user.id, first=100
+            )
+        ]
         for channel in self.users:
             followers = await self.twitch.get_channel_followers(channel.id)
-            stream = await first(self.twitch.get_streams(user_id=[channel.id], first=1))
+            stream = next((s for s in streams if s.user_id == channel.id), None)
+            follow = next((f for f in follows if f.broadcaster_id == channel.id), None)
             sub: UserSubscription | None = None
-            follows: FollowedChannelsResult | None = None
             try:
                 sub = await self.twitch.check_user_subscription(
                     user_id=self.current_user.id, broadcaster_id=channel.id
@@ -96,10 +108,7 @@ async def _async_update_data(self) -> dict[str, TwitchUpdate]:
                 LOGGER.debug("User is not subscribed to %s", channel.display_name)
             except TwitchAPIException as exc:
                 LOGGER.error("Error response on check_user_subscription: %s", exc)
-            else:
-                follows = await self.twitch.get_followed_channels(
-                    self.current_user.id, broadcaster_id=channel.id
-                )
+
             data[channel.id] = TwitchUpdate(
                 channel.display_name,
                 followers.total,
@@ -110,11 +119,11 @@ async def _async_update_data(self) -> dict[str, TwitchUpdate]:
                 stream.started_at if stream else None,
                 stream.thumbnail_url if stream else None,
                 channel.profile_image_url,
-                sub is not None if sub else None,
+                bool(sub),
                 sub.is_gift if sub else None,
                 {"1000": 1, "2000": 2, "3000": 3}.get(sub.tier) if sub else None,
-                follows is not None and follows.total > 0,
-                follows.data[0].followed_at if follows and follows.total else None,
+                bool(follow),
+                follow.followed_at if follow else None,
                 stream.viewer_count if stream else None,
             )
         return data
diff --git a/tests/components/twitch/conftest.py b/tests/components/twitch/conftest.py
index 25e443c2778ae..07732de1b0c69 100644
--- a/tests/components/twitch/conftest.py
+++ b/tests/components/twitch/conftest.py
@@ -111,8 +111,8 @@ def twitch_mock() -> Generator[AsyncMock]:
         mock_client.return_value.get_followed_channels.return_value = TwitchIterObject(
             "get_followed_channels.json", FollowedChannel
         )
-        mock_client.return_value.get_streams.return_value = get_generator(
-            "get_streams.json", Stream
+        mock_client.return_value.get_followed_streams.return_value = get_generator(
+            "get_followed_streams.json", Stream
         )
         mock_client.return_value.check_user_subscription.return_value = (
             UserSubscription(
diff --git a/tests/components/twitch/fixtures/get_followed_channels.json b/tests/components/twitch/fixtures/get_followed_channels.json
index 4add7cc0a98ce..990fac390e95b 100644
--- a/tests/components/twitch/fixtures/get_followed_channels.json
+++ b/tests/components/twitch/fixtures/get_followed_channels.json
@@ -1,9 +1,11 @@
 [
   {
+    "broadcaster_id": 123,
     "broadcaster_login": "internetofthings",
     "followed_at": "2023-08-01"
   },
   {
+    "broadcaster_id": 456,
     "broadcaster_login": "homeassistant",
     "followed_at": "2023-08-01"
   }
diff --git a/tests/components/twitch/fixtures/get_streams.json b/tests/components/twitch/fixtures/get_followed_streams.json
similarity index 89%
rename from tests/components/twitch/fixtures/get_streams.json
rename to tests/components/twitch/fixtures/get_followed_streams.json
index 73f6dc1b42ab2..e02c594c4ccca 100644
--- a/tests/components/twitch/fixtures/get_streams.json
+++ b/tests/components/twitch/fixtures/get_followed_streams.json
@@ -1,5 +1,6 @@
 [
   {
+    "user_id": 123,
     "game_name": "Good game",
     "title": "Title",
     "thumbnail_url": "stream-medium.png",
diff --git a/tests/components/twitch/test_sensor.py b/tests/components/twitch/test_sensor.py
index 0f7ea0c33ebcd..613c0919c49ae 100644
--- a/tests/components/twitch/test_sensor.py
+++ b/tests/components/twitch/test_sensor.py
@@ -21,8 +21,8 @@ async def test_offline(
     hass: HomeAssistant, twitch_mock: AsyncMock, config_entry: MockConfigEntry
 ) -> None:
     """Test offline state."""
-    twitch_mock.return_value.get_streams.return_value = get_generator_from_data(
-        [], Stream
+    twitch_mock.return_value.get_followed_streams.return_value = (
+        get_generator_from_data([], Stream)
     )
     await setup_integration(hass, config_entry)
 


Code review comments:
diff_hunk: @@ -83,11 +83,23 @@ async def _async_update_data(self) -> dict[str, TwitchUpdate]:
             False,
         )
         data = {}
+        streams: list[Stream] = [
+            s
+            async for s in self.twitch.get_followed_streams(
+                user_id=self.current_user.id, first=100
+            )
+        ]
+        follows: list[FollowedChannel] = [
+            f
+            async for f in await self.twitch.get_followed_channels(
+                user_id=self.current_user.id, first=100
+            )
+        ]
path: homeassistant/components/twitch/coordinator.py
text: @user1:
can we make those a dict comprehension so we don't have to loop over almost the full list for every channel?

@author:
I'm sure I could figure out how (if you have tips, it'd be appreciated), but I don't think I'll be able to get to it until after this weekend.

@user1:
That's fine.

```
streams = {s.stream_id: s async for s in ...}
```
to give some pseudo code. This way we can just do `streams[id]`