Repository: Cog-Creators/Red-DiscordBot

Issue:
Scheduled YouTube streams are treated as non-streams
# Other bugs

#### What were you trying to do?

I scheduled a YouTube stream and started it some time later.

#### What were you expecting to happen?

I expected a stream alert when I started the stream.

#### What actually happened?

There was no alert neither when I made the schedule nor when I actually started the stream. This happens because we check if the stream is actually live when it shows up in RSS feed and if it's not, we treat the scheduled stream as non-stream and no longer check for it.

#### How can we reproduce this issue?

1. Add stream alert for your YouTube channel
2. Schedule a stream and wait until it shows up in the RSS (`https://www.youtube.com/feeds/videos.xml?channel_id=CHANNEL_ID_HERE`)
3. Wait 10 minutes for the bot to check the updated RSS (you can lower the needed waiting time with `[p]streamset timer`)
4. Start stream and wait until it shows up in the RSS
5. Wait 10 minutes for the bot to check the updated RSS (you can lower the needed waiting time with `[p]streamset timer`)
6. Stream alert won't show up

---

There's no perfect way for solving this problem, from discussion with Draper, the possible solutions we came up with are:

a) Post info about the scheduled stream as soon as we get info about it (so after number `3` in repro steps) and add info in embed that it's a scheduled stream, not live stream
b) Save info about the scheduled stream and send the alert at the time when it was supposed to start
c) Same as b) but before sending the alert, check that the stream isn't rescheduled/check that it's live (this will make it use more quota which we might really want to avoid)
d) Do both a) and b) (or a) and c))
e) Something we haven't thought of yet?


Discussion:
Just for wider visibility:

Im personally in favour of D,
Doing both A and B, to be specific.

A and C would profile more accurate thing but would also expanentially increase quota usage, since vast majority if Red users only get a 10K quota, we should  accept that streams in core should only show basic info thats allowed with the current quota queries as the 10K quota can be used very fast if you use audio and stream.

IMHO, I think this leaves room for a 3rd party stream cog where said cog could have a more more indepth detailed embeds which users can "opt-in" to use it if they have the quota allowance and are okay loading this cog (this would be similar to modlog and extendedmodlog by Trusty), because it doesnt make sense having a core cog that the vast majority of Red users cant use properly due to the crappy quota given to users.

To give readers an idea, prior to the latest update, we would use about 600K quota per day per channel,  after the most recent update this has been significantly cut down, but iirc this number still around 600/channel/day, so that only leaves room to 16 channels registered across the whole bot assuming audio is not used for spotify.

So the Red team need to decide whether is worthwhile making and maintaining a complex/in-depth cog that most users cant use due to quota limitations, or make a simpler **core** cog that most users can use and delegate a in-depth/detailed cod to the community.
I personally like option D.

Maybe it would be best to offer a switch between A and B or A and C which could also warn the end user about quota limits being reached more easily.

No matter what part of D gets chosen, I think an option to switch off A is important for users that only want notifications when the stream actually goes live.
Looks like this has been forgotten a bit and has the discussion label for way too long. Since this needs to finally move forward, I'm gonna be welcoming PRs for this in a week if I don't get any more feedback here.

**Here's how I see this should be approached based on current feedback:**

When Streams cog sees a new video in the channel's RSS feed, it should first check for the existence of `actualStartTime` and if it doesn't exist, check for `scheduledStartTime` and save it (as well as send the message, if the new setting for sending alerts about scheduled streams is enabled).
This way, Streams cog can be edited to treat the video as a live stream and report the channel as online, when the scheduled start time is reached.

As for:
> Maybe it would be best to offer a switch between A and B or A and C which could also warn the end user about quota limits being reached more easily.

I think it might not be a bad idea, but since it adds on complexity, IMO we should not be doing this until we get feedback suggesting that this is actually needed.
YT Stream notifs are still pretty much useless :(

Resolving patch:
diff --git a/redbot/cogs/streams/streams.py b/redbot/cogs/streams/streams.py
--- a/redbot/cogs/streams/streams.py
+++ b/redbot/cogs/streams/streams.py
@@ -58,6 +58,7 @@ class Streams(commands.Cog):
         "live_message_mention": False,
         "live_message_nomention": False,
         "ignore_reruns": False,
+        "ignore_schedule": False,
     }
 
     role_defaults = {"mention": False}
@@ -223,9 +224,9 @@ async def youtubestream(self, ctx: commands.Context, channel_id_or_name: str):
         apikey = await self.bot.get_shared_api_tokens("youtube")
         is_name = self.check_name_or_id(channel_id_or_name)
         if is_name:
-            stream = YoutubeStream(name=channel_id_or_name, token=apikey)
+            stream = YoutubeStream(name=channel_id_or_name, token=apikey, config=self.config)
         else:
-            stream = YoutubeStream(id=channel_id_or_name, token=apikey)
+            stream = YoutubeStream(id=channel_id_or_name, token=apikey, config=self.config)
         await self.check_online(ctx, stream)
 
     @commands.command()
@@ -398,7 +399,7 @@ async def stream_alert(self, ctx: commands.Context, _class, channel_name):
             is_yt = _class.__name__ == "YoutubeStream"
             is_twitch = _class.__name__ == "TwitchStream"
             if is_yt and not self.check_name_or_id(channel_name):
-                stream = _class(id=channel_name, token=token)
+                stream = _class(id=channel_name, token=token, config=self.config)
             elif is_twitch:
                 await self.maybe_renew_twitch_bearer_token()
                 stream = _class(
@@ -645,6 +646,19 @@ async def ignore_reruns(self, ctx: commands.Context):
             await self.config.guild(guild).ignore_reruns.set(True)
             await ctx.send(_("Streams of type 'rerun' will no longer send an alert."))
 
+    @streamset.command(name="ignoreschedule")
+    @commands.guild_only()
+    async def ignore_schedule(self, ctx: commands.Context):
+        """Toggle excluding YouTube streams schedules from alerts."""
+        guild = ctx.guild
+        current_setting = await self.config.guild(guild).ignore_schedule()
+        if current_setting:
+            await self.config.guild(guild).ignore_schedule.set(False)
+            await ctx.send(_("Streams schedules will be included in alerts."))
+        else:
+            await self.config.guild(guild).ignore_schedule.set(True)
+            await ctx.send(_("Streams schedules will no longer send an alert."))
+
     async def add_or_remove(self, ctx: commands.Context, stream):
         if ctx.channel.id not in stream.channels:
             stream.channels.append(ctx.channel.id)
@@ -705,6 +719,16 @@ async def _stream_alerts(self):
                 pass
             await asyncio.sleep(await self.config.refresh_timer())
 
+    async def _send_stream_alert(
+        self, stream, channel: discord.TextChannel, embed: discord.Embed, content: str = None
+    ):
+        m = await channel.send(
+            content,
+            embed=embed,
+            allowed_mentions=discord.AllowedMentions(roles=True, everyone=True),
+        )
+        stream._messages_cache.append(m)
+
     async def check_streams(self):
         for stream in self.streams:
             with contextlib.suppress(Exception):
@@ -712,9 +736,12 @@ async def check_streams(self):
                     if stream.__class__.__name__ == "TwitchStream":
                         await self.maybe_renew_twitch_bearer_token()
                         embed, is_rerun = await stream.is_online()
+                    elif stream.__class__.__name__ == "YoutubeStream":
+                        embed, is_schedule = await stream.is_online()
                     else:
                         embed = await stream.is_online()
                         is_rerun = False
+                        is_schedule = False
                 except OfflineStream:
                     if not stream._messages_cache:
                         continue
@@ -739,7 +766,14 @@ async def check_streams(self):
                         ignore_reruns = await self.config.guild(channel.guild).ignore_reruns()
                         if ignore_reruns and is_rerun:
                             continue
-
+                        ignore_schedules = await self.config.guild(channel.guild).ignore_schedule()
+                        if ignore_schedules and is_schedule:
+                            continue
+                        if is_schedule:
+                            # skip messages and mentions
+                            await self._send_stream_alert(stream, channel, embed)
+                            await self.save_streams()
+                            continue
                         await set_contextual_locales_from_guild(self.bot, channel.guild)
 
                         mention_str, edited_roles = await self._get_mention_str(
@@ -780,13 +814,7 @@ async def check_streams(self):
                                         str(stream.name), mass_mentions=True, formatting=True
                                     )
                                 )
-
-                        m = await channel.send(
-                            content,
-                            embed=embed,
-                            allowed_mentions=discord.AllowedMentions(roles=True, everyone=True),
-                        )
-                        stream._messages_cache.append(m)
+                        await self._send_stream_alert(stream, channel, embed, content)
                         if edited_roles:
                             for role in edited_roles:
                                 await role.edit(mentionable=False)
@@ -855,6 +883,8 @@ async def load_streams(self):
                     raw_stream["token"] = token.get("client_id")
                     raw_stream["bearer"] = self.ttv_bearer_cache.get("access_token", None)
                 else:
+                    if _class.__name__ == "YoutubeStream":
+                        raw_stream["config"] = self.config
                     raw_stream["token"] = token
             streams.append(_class(**raw_stream))
 
diff --git a/redbot/cogs/streams/streamtypes.py b/redbot/cogs/streams/streamtypes.py
--- a/redbot/cogs/streams/streamtypes.py
+++ b/redbot/cogs/streams/streamtypes.py
@@ -1,9 +1,12 @@
+import contextlib
 import json
 import logging
+from dateutil.parser import parse as parse_time
 from random import choice
 from string import ascii_letters
+from datetime import datetime, timedelta
 import xml.etree.ElementTree as ET
-from typing import ClassVar, Optional, List
+from typing import ClassVar, Optional, List, Tuple
 
 import aiohttp
 import discord
@@ -17,7 +20,7 @@
     YoutubeQuotaExceeded,
 )
 from redbot.core.i18n import Translator
-from redbot.core.utils.chat_formatting import humanize_number
+from redbot.core.utils.chat_formatting import humanize_number, humanize_timedelta
 
 TWITCH_BASE_URL = "https://api.twitch.tv"
 TWITCH_ID_ENDPOINT = TWITCH_BASE_URL + "/helix/users"
@@ -86,6 +89,7 @@ class YoutubeStream(Stream):
     def __init__(self, **kwargs):
         self.id = kwargs.pop("id", None)
         self._token = kwargs.pop("token", None)
+        self._config = kwargs.pop("config")
         self.not_livestreams: List[str] = []
         self.livestreams: List[str] = []
 
@@ -128,8 +132,11 @@ async def is_online(self):
                     if (
                         stream_data
                         and stream_data != "None"
-                        and stream_data.get("actualStartTime", None) is not None
                         and stream_data.get("actualEndTime", None) is None
+                        and (
+                            stream_data.get("actualStartTime", None) is not None
+                            or stream_data.get("scheduledStartTime", None) is not None
+                        )
                     ):
                         if video_id not in self.livestreams:
                             self.livestreams.append(data["items"][0]["id"])
@@ -143,24 +150,52 @@ async def is_online(self):
         # info from the RSS ... but incase you don't wanna deal with fully rewritting the
         # code for this part, as this is only a 2 quota query.
         if self.livestreams:
-            params = {"key": self._token["api_key"], "id": self.livestreams[-1], "part": "snippet"}
+            params = {
+                "key": self._token["api_key"],
+                "id": self.livestreams[-1],
+                "part": "snippet,liveStreamingDetails",
+            }
             async with aiohttp.ClientSession() as session:
                 async with session.get(YOUTUBE_VIDEOS_ENDPOINT, params=params) as r:
                     data = await r.json()
-            return self.make_embed(data)
+            return await self.make_embed(data)
         raise OfflineStream()
 
-    def make_embed(self, data):
+    async def make_embed(self, data):
         vid_data = data["items"][0]
         video_url = "https://youtube.com/watch?v={}".format(vid_data["id"])
         title = vid_data["snippet"]["title"]
         thumbnail = vid_data["snippet"]["thumbnails"]["medium"]["url"]
         channel_title = vid_data["snippet"]["channelTitle"]
         embed = discord.Embed(title=title, url=video_url)
+        is_schedule = False
+        if vid_data["liveStreamingDetails"]["scheduledStartTime"] is not None:
+            if "actualStartTime" not in vid_data["liveStreamingDetails"]:
+                start_time = parse_time(vid_data["liveStreamingDetails"]["scheduledStartTime"])
+                start_in = start_time.replace(tzinfo=None) - datetime.now()
+                embed.description = _("This stream will start in {time}").format(
+                    time=humanize_timedelta(
+                        timedelta=timedelta(minutes=start_in.total_seconds() // 60)
+                    )  # getting rid of seconds
+                )
+                embed.timestamp = start_time
+                is_schedule = True
+            else:
+                # repost message
+                to_remove = []
+                for message in self._messages_cache:
+                    if message.embeds[0].description is discord.Embed.Empty:
+                        continue
+                    with contextlib.suppress(Exception):
+                        autodelete = await self._config.guild(message.guild).autodelete()
+                        if autodelete:
+                            await message.delete()
+                    to_remove.append(message.id)
+                self._messages_cache = [x for x in self._messages_cache if x.id not in to_remove]
         embed.set_author(name=channel_title)
         embed.set_image(url=rnd(thumbnail))
         embed.colour = 0x9255A5
-        return embed
+        return embed, is_schedule
 
     async def fetch_id(self):
         return await self._fetch_channel_resource("id")


Test patch: