Repository: Cog-Creators/Red-DiscordBot

Issue:
[Streams] YouTube streams are wiped when RSS query returns HTTP 404
### What Red version are you using?

3.4.12

### What were you trying to do?

Add YouTube streams, and then wait for YouTube live stream notifications.

### What did you expect to happen?

Get live stream notifications.

### What actually happened?

Got live stream notifications for a few days, and then noticed nothing happened. After using the `[p]streamalert list` command, I noticed all YouTube streams were removed.

I re-added the channels again the first time this happened, and also added more logging (see diff below) to try to understand it better. A few days later, the same thing happened again.

### How can we reproduce this error?

Unsure at the moment.
1. Add YouTube channels that stream regularly.
2. Wait until streams don't show up.

### Anything else?

FWIW, I added a few debug logging statements to help try to understand it better. Diff of changes to add the logging statements (I probably should've used `stack_info=True` instead):
```
(renchan) ubuntu@LuiV-Eldridge:~/github/renchan/Ren/redbot/cogs/streams$ git diff streamtypes.py
diff --git a/redbot/cogs/streams/streamtypes.py b/redbot/cogs/streams/streamtypes.py
index 294f2c7d..d229eccb 100644
--- a/redbot/cogs/streams/streamtypes.py
+++ b/redbot/cogs/streams/streamtypes.py
@@ -126,6 +126,8 @@ class YoutubeStream(Stream):
         async with aiohttp.ClientSession() as session:
             async with session.get(YOUTUBE_CHANNEL_RSS.format(channel_id=self.id)) as r:
                 if r.status == 404:
+                    log.debug( "Could not find stream, here's some more info to help: %s",
+                               r, exc_info=True)
                     raise StreamNotFound()
                 rssdata = await r.text()

@@ -269,6 +271,8 @@ class YoutubeStream(Stream):

         self._check_api_errors(data)
         if "items" in data and len(data["items"]) == 0:
+            log.debug( "Could not find stream in this if, here's some more info to help: %s",
+                       data, exc_info=True)
             raise StreamNotFound()
         elif "items" in data:
             return data["items"][0][resource]
@@ -277,6 +281,8 @@ class YoutubeStream(Stream):
             and "totalResults" in data["pageInfo"]
             and data["pageInfo"]["totalResults"] < 1
         ):
+            log.debug( "Could not find stream in this elif, here's some more info to help: %s",
+                       data, exc_info=True)
             raise StreamNotFound()
         raise APIError(r.status, data)
```

```
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCdJP7RSYTkj5KofYQqSDgNA) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:48:37 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'S
AMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:48:37] [INFO] red.core.cogs.Streams: Stream with name Lily Lunaire no longer exists. Removing...
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: RGsmcgWw7Z8
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: 9mj2QJE-UOI
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: ICim2O8u2ms
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: eo0nu3pKgc8
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: S_M4CJgJDvk
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: HQ0JX8SD-5E
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: LButyx65snY
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: CNN0QKp08Wg
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: o2a_zc6-71Y
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: 711J_W0-qfY
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: W0Fo39SpN-g
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: bDN9KiWdXbs
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: yitzIgzLLWY
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: yeXlL3w-NRI
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: CCeuVUZ7JJc
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: livestreams for Akaihaato Ch. 赤井はあと: []
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: not_livestreams for Akaihaato Ch. 赤井はあと: ['RGsmcgWw7Z8', '9mj2QJE-UOI', 'ICim2O8u2ms', 'eo0nu3pKgc8', 'S_M4CJgJDvk', 'HQ0JX8SD-5E', 'LButyx65snY', 'CNN0QKp08Wg', 'o2a_zc6-71Y', '711J_W0-qfY', 'W0Fo39SpN-g', 'bDN9KiWdXbs', 'yitzIgzLLWY', 'yeXlL3w-NRI', 'CCeuVUZ7JJc']
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: A2QGP7rhHaU
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: yQ3lRlQm_A0
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: -8fcR-G393U
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: 5QZa0nEyGQU
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: PaX_AURtDRo
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: rwHLXMESSyY
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: nVyr5cNCxcA
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: SZln3voSl3s
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: YyR_PLyxIzM
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: xsLJ65DTjJc
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: IgVauHBfYDc
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: m_KIu6Sbpjk
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: b4Wi303Epv0
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: nuYoSAWxs6k
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: video_id in not_livestreams: KHgXn6j9w1g
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: livestreams for Sachi Saiki 才気桜千 Channel: []
[2021-07-22 23:48:37] [DEBUG] red.core.cogs.Streams: not_livestreams for Sachi Saiki 才気桜千 Channel: ['-8fcR-G393U', '5QZa0nEyGQU', 'PaX_AURtDRo', 'rwHLXMESSyY', 'nVyr5cNCxcA', 'SZln3voSl3s', 'YyR_PLyxIzM', 'xsLJ65DTjJc', 'IgVauHBfYDc', 'm_KIu6Sbpjk', 'b4Wi303Epv0', 'nuYoSAWxs6k', 'KHgXn6j9w1g', 'c61a6_Usoq8', 'm1gzizK6u68', 'yQ3lRlQm_A0', 'A2QGP7rhHaU']

<truncated>

[2021-07-22 23:51:40] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UC1CfXB_kRs3C-zaeTG3oGyg) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:40 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:40] [INFO] red.core.cogs.Streams: Stream with name Akaihaato Ch. 赤井はあと no longer exists. Removing...
[2021-07-22 23:51:40] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCZGrC7taxf_O4WMsTj__aYA) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:40 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:40] [INFO] red.core.cogs.Streams: Stream with name Sachi Saiki 才気桜千 Channel no longer exists. Removing...
[2021-07-22 23:51:40] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCHsx4Hqa-1ORjQTh9TYDhww) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:40 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:40] [INFO] red.core.cogs.Streams: Stream with name Takanashi Kiara Ch. hololive-EN no longer exists. Removing...
[2021-07-22 23:51:41] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCL_qhgtOy0dy1Agp8vkySQg) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:41 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:41] [INFO] red.core.cogs.Streams: Stream with name Mori Calliope Ch. hololive-EN no longer exists. Removing...
[2021-07-22 23:51:41] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCoSrY_IQQVpmIRZ9Xf-y93g) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:41 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:41] [INFO] red.core.cogs.Streams: Stream with name Gawr Gura Ch. hololive-EN no longer exists. Removing...
[2021-07-22 23:51:41] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCyl1z3jo3XHR1riLFKG5UAg) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:41 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:41] [INFO] red.core.cogs.Streams: Stream with name Watson Amelia Ch. hololive-EN no longer exists. Removing...
[2021-07-22 23:51:41] [DEBUG] red.core.cogs.Streams: Could not find stream, here's some more info to help: <ClientResponse(https://www.youtube.com/feeds/videos.xml?channel_id=UCMwGHR0BTZuLsmjY_NT5Pwg) [404 Not Found]>
<CIMultiDictProxy('Date': 'Fri, 23 Jul 2021 06:51:41 GMT', 'Content-Type': 'text/html; charset=UTF-8', 'Server': 'YouTube RSS Feeds server', 'Content-Length': '1613', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"')>
NoneType: None
[2021-07-22 23:51:41] [INFO] red.core.cogs.Streams: Stream with name Ninomae Ina'nis Ch. hololive-EN no longer exists. Removing...
[2021-07-22 23:51:48] [DEBUG] red.cogs.mutes: Cleaning unmute tasks
```

Not sure what would be a good way to resolve this, but since we check every few minutes (or whatever we set it to, I don't recall), if we fail x number of times in a row, then we could remove it (as opposed to removing it right away when raising `StreamNotFound`).


Discussion:
Second on this issue.  It stays and then poofs, which causes Live Alerts to be missed

Resolving patch:
diff --git a/redbot/cogs/streams/streams.py b/redbot/cogs/streams/streams.py
--- a/redbot/cogs/streams/streams.py
+++ b/redbot/cogs/streams/streams.py
@@ -31,6 +31,8 @@
 from collections import defaultdict
 from typing import Optional, List, Tuple, Union, Dict
 
+MAX_RETRY_COUNT = 10
+
 _ = Translator("Streams", __file__)
 log = logging.getLogger("red.core.cogs.Streams")
 
@@ -747,8 +749,14 @@ async def check_streams(self):
                     else:
                         embed = await stream.is_online()
                 except StreamNotFound:
-                    log.info("Stream with name %s no longer exists. Removing...", stream.name)
-                    to_remove.append(stream)
+                    if stream.retry_count > MAX_RETRY_COUNT:
+                        log.info("Stream with name %s no longer exists. Removing...", stream.name)
+                        to_remove.append(stream)
+                    else:
+                        log.info(
+                            "Stream with name %s seems to not exist, will retry later", stream.name
+                        )
+                        stream.retry_count += 1
                     continue
                 except OfflineStream:
                     if not stream.messages:
diff --git a/redbot/cogs/streams/streamtypes.py b/redbot/cogs/streams/streamtypes.py
--- a/redbot/cogs/streams/streamtypes.py
+++ b/redbot/cogs/streams/streamtypes.py
@@ -64,6 +64,9 @@ def __init__(self, **kwargs):
         # self.already_online = kwargs.pop("already_online", False)
         self.messages = kwargs.pop("messages", [])
         self.type = self.__class__.__name__
+        # Keep track of how many failed consecutive attempts we had at checking
+        # if the stream's channel actually exists.
+        self.retry_count = 0
 
     @property
     def display_name(self) -> Optional[str]:
@@ -129,6 +132,10 @@ async def is_online(self):
                     raise StreamNotFound()
                 rssdata = await r.text()
 
+        # Reset the retry count since we successfully got information about this
+        # channel's streams
+        self.retry_count = 0
+
         if self.not_livestreams:
             self.not_livestreams = list(dict.fromkeys(self.not_livestreams))
 
@@ -401,6 +408,10 @@ async def is_online(self):
             if follows_data:
                 final_data["followers"] = follows_data["total"]
 
+            # Reset the retry count since we successfully got information about this
+            # channel's streams
+            self.retry_count = 0
+
             return self.make_embed(final_data), final_data["type"] == "rerun"
         elif stream_code == 400:
             raise InvalidTwitchCredentials()
@@ -462,6 +473,9 @@ async def is_online(self):
                 data = await r.text(encoding="utf-8")
         if r.status == 200:
             data = json.loads(data)
+            # Reset the retry count since we successfully got information about this
+            # channel's streams
+            self.retry_count = 0
             if data["online"] is True:
                 # self.already_online = True
                 return self.make_embed(data)


Test patch: