Repository: python-discord/bot

Issue:
The Reddit cog occassionally fails to relay new /r/python messages
Occasionally, the [Reddit cog](/python-discord/bot/blob/master/bot/cogs/reddit.py) stops relaying new messages to the #reddit channel. From what I've seen, there are two ways in which the cog breaks and both should be easy to fix.

#### 1. The background task fails on unexpected http responses
The background task to fetch new posts, the `self.new_posts_task` created in the `on_ready` event listener, assumes that the response we get from the Reddit API is valid `json` in [this line](/python-discord/bot/blob/master/bot/cogs/reddit.py#L55). If, for some temporary reason, we don't get a `json` response, but a `text/html` response (404, probably), the task fails with the following exception:

```py
In [160]: bot.get_cog("Reddit").new_posts_task
Out[160]: 
  File "/bot/bot/cogs/eval.py", line 167, in _eval
    res = await func()
  File "<string>", line 8, in func
  File "/bot/bot/cogs/eval.py", line 167, in _eval
    res = await func()
  File "<string>", line 8, in func
  File "/bot/bot/cogs/reddit.py", line 142, in poll_new_posts
    posts = await self.fetch_posts(f"{subreddit}/new")
  File "/bot/bot/cogs/reddit.py", line 55, in fetch_posts
    content = await response.json()
  File "/bot/.venv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 938, in json
    headers=self.headers)
aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=utf-8'
```

This line is in the `fetch_posts` utility function used by both the `new_posts_task` as well as the `top_weekly_posts_task`. This means that can break because of this error. 

The likely solution is to either handle the exception or check the response status code before trying to parse it as json. We probably want to either have some kind of retry logic in the handling of the non-200 response, since otherwise the weekly top posts task skips a week.

#### ~~2. The channel can't be initialized during `on_ready`~~

~~Similar to the issue we had with the watch channels, it can occur that the bot's internal channel cache isn't fully loaded when the `on_ready` event fires. This means that the channel will not be retrieved and the tasks will never be started. This only happens intermittently, but it does occasionally happen.~~

~~In such cases, the `self.bot.get_channel` returns `None`, because the channel has not been loaded into the internal bot cache yet at that point.~~

```py
    async def on_ready(self):
        self.reddit_channel = self.bot.get_channel(Channels.reddit)
```

~~While this could be fixed by adding a slight delay or a bit of retry logic, another option is to wait for the migration to a later version of `discord.py` and use `await self.bot.fetch_channel` here instead. That will fetch the channel from the API, bypassing any issues we may have with the internal bot channel cache.~~


Discussion:
Hi! It looks like this hasn't been claimed by anyone yet. Can I take a stab at it? I think it's within my abilities but I'll indicate within a week if I don't think I can handle it for some reason.
The second problem should no longer be a problem; it was fixed during the upgrade to the latest version of discord.py.

Resolving patch:
diff --git a/bot/cogs/reddit.py b/bot/cogs/reddit.py
--- a/bot/cogs/reddit.py
+++ b/bot/cogs/reddit.py
@@ -21,6 +21,7 @@ class Reddit(Cog):
 
     HEADERS = {"User-Agent": "Discord Bot: PythonDiscord (https://pythondiscord.com/)"}
     URL = "https://www.reddit.com"
+    MAX_FETCH_RETRIES = 3
 
     def __init__(self, bot: Bot):
         self.bot = bot
@@ -42,16 +43,23 @@ async def fetch_posts(self, route: str, *, amount: int = 25, params: dict = None
         if params is None:
             params = {}
 
-        response = await self.bot.http_session.get(
-            url=f"{self.URL}/{route}.json",
-            headers=self.HEADERS,
-            params=params
-        )
+        url = f"{self.URL}/{route}.json"
+        for _ in range(self.MAX_FETCH_RETRIES):
+            response = await self.bot.http_session.get(
+                url=url,
+                headers=self.HEADERS,
+                params=params
+            )
+            if response.status == 200 and response.content_type == 'application/json':
+                # Got appropriate response - process and return.
+                content = await response.json()
+                posts = content["data"]["children"]
+                return posts[:amount]
 
-        content = await response.json()
-        posts = content["data"]["children"]
+            await asyncio.sleep(3)
 
-        return posts[:amount]
+        log.debug(f"Invalid response from: {url} - status code {response.status}, mimetype {response.content_type}")
+        return list()  # Failed to get appropriate response within allowed number of retries.
 
     async def send_top_posts(
         self, channel: TextChannel, subreddit: Subreddit, content: str = None, time: str = "all"
@@ -62,13 +70,14 @@ async def send_top_posts(
         embed.description = ""
 
         # Get the posts
-        posts = await self.fetch_posts(
-            route=f"{subreddit}/top",
-            amount=5,
-            params={
-                "t": time
-            }
-        )
+        async with channel.typing():
+            posts = await self.fetch_posts(
+                route=f"{subreddit}/top",
+                amount=5,
+                params={
+                    "t": time
+                }
+            )
 
         if not posts:
             embed.title = random.choice(ERROR_REPLIES)


Test patch: