Repository: praw-dev/praw

Issue:
ClientException: Invalid URL for Random Submission
## Issue Description

For some subreddits an Invalid URL exception is thrown when I try to get a random submission.

Code:
```python
import praw
from config.reddit import client

reddit = praw.Reddit(client_id = client['id'],
    client_secret = client['secret'],
    user_agent = client['agent'])

print(reddit.subreddit('pics').random()) # works
print(reddit.subreddit('wallpapers').random()) # does not work
```

Output:
```
7s53b4
Traceback (most recent call last):
  File "./test.py", line 9, in <module>
    print(reddit.subreddit('wallpapers').random()) # does not work
  File "/usr/local/lib/python3.6/site-packages/praw/models/reddit/subreddit.py", line 339, in random
    self._reddit.config.reddit_url, path))
  File "/usr/local/lib/python3.6/site-packages/praw/models/reddit/submission.py", line 125, in __init__
    self.id = self.id_from_url(url)
  File "/usr/local/lib/python3.6/site-packages/praw/models/reddit/submission.py", line 41, in id_from_url
    raise ClientException('Invalid URL: {}'.format(url))
praw.exceptions.ClientException: Invalid URL: https://www.reddit.com/r/wallpapers/
```

## System Information

PRAW Version: 5.3.0
Python Version: 3.6.4
Operating System: Windows and Linux



Discussion:
Looking into this, I've found some interesting things.

I notice this issue in web-based Reddit as well, not just PRAW. Loading https://reddit.com/r/pics/random in the browser redirects to a random post as expected, but https://reddit.com/r/wallpapers/random, redirects to the front page of the subreddit. 

There doesn't seem to be a standard option to disable /random in subreddit settings. This makes me wonder why /r/wallpapers seems to have this unique behavior. Nothing in the vars of the two subreddits seemed to explain the difference.

Are there any other subreddits that you know of that don't properly give a random post?

The exception in PRAW is caused by Reddit trying to redirect /r/wallpapers/random back to /r/wallpapers, when PRAW expects a submission, not a subreddit. PRAW 5.3.0 tries to parse the url and fails. Something else happens in 5.4.0.dev0 (see below).

To me, this comes off as an undocumented issue on Reddit's end. I can't see why Reddit should be redirecting in this way. I don't know how PRAW should handle it — thoughts @bboe?

---

An interesting note: I tested this bug on both PRAW 5.3.0 and the latest dev version (9d9f17822d2d970d1e06fdf69c56ba82a537655b), and the behavior is different for the two. In the dev version, when I run `print(repr(reddit.subreddit('wallpapers').random()))`,  the output is `Submission(id='wallpapers')`. As of the newest version, PRAW creates a `Submission` object with no complaints (until you try to access its attributes). This is because of b27047be7ce131f31cff85e1a67bd0259450f1a9, which has the side effect of accepting "https://reddit.com/r/wallpapers/" as a valid submission URL (until network requests are made with that ID). I'll leave it up to @bboe if that's a bad thing.
@jarhill0 you're so amazing. I was just looking into this and bam I see your comment get added.

I would consider the behavior introduced in the dev version a bug. Such a URL should not be accepted and should continue to raise the same exception. Do you think you could make a PR to make `id_from_url` a little more specific?

@Patreek87 thanks for filing. Given that this issue affects Reddit and will resolve itself once Reddit behaves the way it should, I'd actually say this is a non-issue. While there might have been some odd work-arounds in older versions of PRAW, PRAW4+ doesn't try to do anything special to work around bugs that exist on Reddit's end.

Consider making a post on /r/bugs that random doesn't work on some subreddits. If we learn that this is expected behavior, then at such a time we can consider making a special exception for such subreddits.
**TLDR**: This is a side effect of excluding a subreddit from /r/all. As far as I can tell, there is no way to tell whether a subreddit is excluded from /r/all.

---

I did further research and found [this thread](https://www.reddit.com/r/bugs/comments/4qwt3j/rawwnimerandom_redirects_to_rawwnime_not_a_random/). 

/u/13steinj points out that Reddit source is designed to do this when a subreddit's moderators do not check the box "allow this subreddit to be exposed to users in /r/all, /r/popular, default, and trending lists" in their subreddit settings.

To me (and to /u/GoldenSights in that thread) this seems like a mistake. Yes, it's coded that way, but it seems like the intention was to prevent quarantined and non-all subreddits from being included in /r/random. Instead, unchecking that box (or being quarantined) means that https://reddit.com/r/{subreddit}/random will not work. 

@madbook made the commits in Reddit, so I'm tagging him in hopes that he can help us figure this out. He said in the thread linked above:

> […] sounds likely that excluding from r/subreddit/random is an unintentional side effect of excluding (intentionally) from r/random.

So Reddit would call this expected behavior, but I don't think it is, from the user's perspective. There is no reason that https://reddit.com/r/{subreddit}/random should not work for quarantined or non-all subreddits. After all, a user going to that URL will already know of that subreddit.

This doesn't seem like something that would be too difficult to change (but I have no experience with Reddit's codebase, so I'm just speculating). Ideally a change could be implemented at Reddit so that this feature works in all subreddits. 

If it can't be implemented, a potential workaround would be to raise an exception if `.random()` is called on a subreddit that doesn't support it. If there exists an attribute that reveals whether the "exclude from /r/all" box is ticked, we could use that to determine whether a subreddit is compatible with `.random()`. Unfortunately, there seems to be no such attribute. `allow_discovery` sounds appropriate, but it is `True` for both r/pics and r/wallpapers:

```
In [8]: reddit.subreddit('pics').allow_discovery
Out[8]: True

In [9]: reddit.subreddit('wallpapers').allow_discovery
Out[9]: True

```

At this stage, the only action I believe PRAW could take would be to perform validation on the redirect here: https://github.com/praw-dev/praw/blob/9da543413d884a8ee0e1cf2450b48961268c8076/praw/models/reddit/subreddit.py#L336-L341

For now, it's probably better to wait for one of the following to happen:

- Someone at Reddit says that `allow_discovery` was improperly implemented, fixes it, and we use that as a workaround
- Someone at Reddit decides to modify the behavior of https://reddit.com/r/{subreddit}/random to work for all active subreddits (my personal preferred outcome)
- Someone at Reddit confirms that they will not change either of the above features
- Someone points out that I missed the attribute that reveals whether a subreddit is excluded from all, and we use that as a workaround

The issue is out of our control, and while there is one way of trying to work around it, there are possible better outcomes.

---

Sorry for the massive wall of text. Here's the TLDR from the top again: 

**TLDR**: This is a side effect of excluding a subreddit from /r/all. As far as I can tell, there is no way to tell whether a subreddit is excluded from /r/all.
Re-opening given the discussion that this might be intentional behavior on Reddit's end.

> As far as I can tell, there is no way to tell whether a subreddit is excluded from /r/all.

It seems like this _is_ the way to tell if a subreddit is excluded from /r/all. Even if there were such a setting, it's not worth it to check since we get the feedback anyway. We could simply handle the ClientException for redirect inside `random` and raise a different exception in such cases. It would be good, however, to first get confirmation from Reddit that this is not actually a bug.
Hey @jarhill0, thanks for the ping.  I think this isn't a bug _per se_, but I totally agree that it's not great behavior.  It's been a while since I've looked at that code, but IIRC r/random, r/<subreddit>/random, r/all, search, and maybe one or two other small things were all tied to the same subreddit preference.  At some point later a separate setting was added, `allow_discovery`, which some of those features are now tied to (primarily search, I think).

I don't _think_ `allow_discovery` is actually broken, it just doesn't currently control the random features.  It's still controlled by the old setting, which isn't available in the API response. I tend to agree that r/<subreddit>/random should probably just work regardless of either setting, but I can't commit to changing it without running it by other folks internally and triaging it against all the other work we're doing.  That said, I'll bring it up and see if it's something that we can squeeze in.

I appreciate the response! Thanks for looking into it. 
@madbook is the behavior of /r/random tied to the behavior of /r/{SUBREDDIT}/random? We're more concerned with the latter here. Thanks!
@bboe IIRC, the two behaviors _are_ currently connected.  If any change comes about from this it should only affect the behavior of `/r/{SUBREDDIT}/random`.

Resolving patch:
diff --git a/praw/models/reddit/subreddit.py b/praw/models/reddit/subreddit.py
--- a/praw/models/reddit/subreddit.py
+++ b/praw/models/reddit/subreddit.py
@@ -527,15 +527,22 @@ def _upload_media(self, media_path):
         return upload_url + "/" + upload_data["key"]
 
     def random(self):
-        """Return a random Submission."""
+        """Return a random Submission.
+
+        Returns ``None`` on subreddits that do not support the random feature.
+        One example, at the time of writing, is /r/wallpapers.
+        """
         url = API_PATH["subreddit_random"].format(subreddit=self)
         try:
             self._reddit.get(url, params={"unique": self._reddit._next_unique})
         except Redirect as redirect:
             path = redirect.path
-        return self._submission_class(
-            self._reddit, url=urljoin(self._reddit.config.reddit_url, path)
-        )
+        try:
+            return self._submission_class(
+                self._reddit, url=urljoin(self._reddit.config.reddit_url, path)
+            )
+        except ClientException:
+            return None
 
     def rules(self):
         """Return rules for the subreddit.
diff --git a/pre_push.py b/pre_push.py
--- a/pre_push.py
+++ b/pre_push.py
@@ -31,7 +31,7 @@ def main():
     """Entry point to pre_push.py."""
     success = True
     success &= do_process(["black *.py docs praw tests"], shell=True)
-    success &= do_process(["flake8", "--exclude=.eggs,docs,.tox"])
+    success &= do_process(["flake8", "--exclude=.eggs,build,dist,docs,.tox"])
     success &= do_process(["pydocstyle", "praw"])
     # success &= do_process(["pylint", "--rcfile=.pylintrc", "praw"])
 


Test patch:
diff --git a/tests/integration/cassettes/TestSubreddit.test_random__returns_none.json b/tests/integration/cassettes/TestSubreddit.test_random__returns_none.json
new file mode 100644
--- /dev/null
+++ b/tests/integration/cassettes/TestSubreddit.test_random__returns_none.json
@@ -0,0 +1,220 @@
+{
+  "http_interactions": [
+    {
+      "recorded_at": "2019-03-09T18:26:01",
+      "request": {
+        "body": {
+          "encoding": "utf-8",
+          "string": "grant_type=client_credentials"
+        },
+        "headers": {
+          "Accept": [
+            "*/*"
+          ],
+          "Accept-Encoding": [
+            "identity"
+          ],
+          "Authorization": [
+            "Basic <BASIC_AUTH>"
+          ],
+          "Connection": [
+            "keep-alive"
+          ],
+          "Content-Length": [
+            "29"
+          ],
+          "Content-Type": [
+            "application/x-www-form-urlencoded"
+          ],
+          "User-Agent": [
+            "<USER_AGENT> PRAW/6.1.2.dev0 prawcore/1.0.1"
+          ]
+        },
+        "method": "POST",
+        "uri": "https://www.reddit.com/api/v1/access_token"
+      },
+      "response": {
+        "body": {
+          "encoding": "UTF-8",
+          "string": "{\"access_token\": \"<ACCESS_TOKEN>\", \"token_type\": \"bearer\", \"expires_in\": 3600, \"scope\": \"*\"}"
+        },
+        "headers": {
+          "Accept-Ranges": [
+            "bytes"
+          ],
+          "Connection": [
+            "keep-alive"
+          ],
+          "Content-Length": [
+            "106"
+          ],
+          "Content-Type": [
+            "application/json; charset=UTF-8"
+          ],
+          "Date": [
+            "Sat, 09 Mar 2019 18:26:03 GMT"
+          ],
+          "Server": [
+            "snooserv"
+          ],
+          "Set-Cookie": [
+            "edgebucket=GwXc0wbGP8qmtmR16B; Domain=reddit.com; Max-Age=63071999; Path=/;  secure"
+          ],
+          "Strict-Transport-Security": [
+            "max-age=15552000; includeSubDomains; preload"
+          ],
+          "Via": [
+            "1.1 varnish"
+          ],
+          "X-Cache": [
+            "MISS"
+          ],
+          "X-Cache-Hits": [
+            "0"
+          ],
+          "X-Moose": [
+            "majestic"
+          ],
+          "X-Served-By": [
+            "cache-bur17520-BUR"
+          ],
+          "X-Timer": [
+            "S1552155963.947519,VS0,VE334"
+          ],
+          "cache-control": [
+            "max-age=0, must-revalidate"
+          ],
+          "x-content-type-options": [
+            "nosniff"
+          ],
+          "x-frame-options": [
+            "SAMEORIGIN"
+          ],
+          "x-reddit-loid": [
+            "00000000003dkbrpbb.2.1552155963235.Z0FBQUFBQmNoQVU3UElPYXZKVU8wRWVPN0w0RXVzV3hWNWRTRFpIRnBuZm9jeFEtWlZ5V25BZEt5d3EtQXoxSkZScnJhRHVjS2RHcVhURjJVMTF5ZVVVYms0RlpSVEFnUjZ5QU5aWUdsUHE0QW9kSXRDVEFtOUhlVC02NVFLbGN4Qk94RkdmakxScl8"
+          ],
+          "x-xss-protection": [
+            "1; mode=block"
+          ]
+        },
+        "status": {
+          "code": 200,
+          "message": "OK"
+        },
+        "url": "https://www.reddit.com/api/v1/access_token"
+      }
+    },
+    {
+      "recorded_at": "2019-03-09T18:26:01",
+      "request": {
+        "body": {
+          "encoding": "utf-8",
+          "string": ""
+        },
+        "headers": {
+          "Accept": [
+            "*/*"
+          ],
+          "Accept-Encoding": [
+            "identity"
+          ],
+          "Authorization": [
+            "bearer <ACCESS_TOKEN>"
+          ],
+          "Connection": [
+            "keep-alive"
+          ],
+          "Cookie": [
+            "edgebucket=GwXc0wbGP8qmtmR16B"
+          ],
+          "User-Agent": [
+            "<USER_AGENT> PRAW/6.1.2.dev0 prawcore/1.0.1"
+          ]
+        },
+        "method": "GET",
+        "uri": "https://oauth.reddit.com/r/wallpapers/random/?unique=0&raw_json=1"
+      },
+      "response": {
+        "body": {
+          "encoding": "UTF-8",
+          "string": ""
+        },
+        "headers": {
+          "Accept-Ranges": [
+            "bytes"
+          ],
+          "Connection": [
+            "keep-alive"
+          ],
+          "Content-Length": [
+            "0"
+          ],
+          "Content-Type": [
+            "application/json; charset=UTF-8"
+          ],
+          "Date": [
+            "Sat, 09 Mar 2019 18:26:03 GMT"
+          ],
+          "Server": [
+            "snooserv"
+          ],
+          "Strict-Transport-Security": [
+            "max-age=15552000; includeSubDomains; preload"
+          ],
+          "Via": [
+            "1.1 varnish"
+          ],
+          "X-Cache": [
+            "MISS"
+          ],
+          "X-Cache-Hits": [
+            "0"
+          ],
+          "X-Moose": [
+            "majestic"
+          ],
+          "X-Served-By": [
+            "cache-bur17528-BUR"
+          ],
+          "X-Timer": [
+            "S1552155963.475956,VS0,VE211"
+          ],
+          "access-control-allow-origin": [
+            "*"
+          ],
+          "access-control-expose-headers": [
+            "X-Moose"
+          ],
+          "cache-control": [
+            "max-age=0, must-revalidate"
+          ],
+          "location": [
+            "https://www.reddit.com/r/wallpapers/.json?utm_campaign=redirect&utm_medium=desktop&utm_source=reddit&utm_name=random_link"
+          ],
+          "set-cookie": [
+            "loid=00000000003dkbrpbb.2.1552155963235.Z0FBQUFBQmNoQVU3RzBWcURXYXBNTzdCU0FEelAyaXc0OHpmTm5vdFh2dUNyYmVHNU4yV0U0WHJSY2pxMTRfT09RMEstT1lRRS1mWmkzVE42aXFEU00xSTdaYTE3X3hDZEttSlh1TnJrVzFnRUdfc0pYcjhVdnB1T3hkYjFTYmJtN0h6ZGc1YnFLUUY; Domain=reddit.com; Max-Age=63071999; Path=/; expires=Mon, 08-Mar-2021 18:26:03 GMT; secure",
+            "session_tracker=ES9DTLwanxM1RT5ULY.0.1552155963543.Z0FBQUFBQmNoQVU3WWwyUXpHMDZqTW5XUG9hdDR1aGJhcmplVkNyT18xQ0F2WlNydFZWeVlmNjdVYUhfSHBuem52eVAzZ3RWcURRMnpHNC1uODhYN1RmeTZSXzRlNFFlQmNHUG1HOVhna2FjZWNVQjZsdW1tRVd2UUFZZjF0ZjJYRVpFcWJlY0FQVGI; Domain=reddit.com; Max-Age=7199; Path=/; expires=Sat, 09-Mar-2019 20:26:03 GMT; secure"
+          ],
+          "x-content-type-options": [
+            "nosniff"
+          ],
+          "x-frame-options": [
+            "SAMEORIGIN"
+          ],
+          "x-ua-compatible": [
+            "IE=edge"
+          ],
+          "x-xss-protection": [
+            "1; mode=block"
+          ]
+        },
+        "status": {
+          "code": 302,
+          "message": "Found"
+        },
+        "url": "https://oauth.reddit.com/r/wallpapers/random/?unique=0&raw_json=1"
+      }
+    }
+  ],
+  "recorded_with": "betamax/0.8.1"
+}
\ No newline at end of file
diff --git a/tests/integration/models/reddit/test_subreddit.py b/tests/integration/models/reddit/test_subreddit.py
--- a/tests/integration/models/reddit/test_subreddit.py
+++ b/tests/integration/models/reddit/test_subreddit.py
@@ -133,7 +133,14 @@ def test_random(self):
                 subreddit.random(),
                 subreddit.random(),
             ]
-            assert len(submissions) == len(set(submissions))
+        assert len(submissions) == len(set(submissions))
+
+    def test_random__returns_none(self):
+        with self.recorder.use_cassette(
+            "TestSubreddit.test_random__returns_none"
+        ):
+            subreddit = self.reddit.subreddit("wallpapers")
+            assert subreddit.random() is None
 
     def test_rules(self):
         with self.recorder.use_cassette("TestSubreddit.test_rules"):