Repository: yt-dlp/yt-dlp

Issue:
Reddit.com (quarantined) Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>)
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that a **supported** site is broken
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Worldwide

### Provide a description that is worded well enough to be understood

Downloading videos from a quarantined reddit subreddit fail, both with and without --cookies-from-browser. Downloading with login credentials are not supported. 

Applies to nightly@2023.04.08.161042 (currently latest)

Example url : https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
PS C:\Users\XXXX\Downloads> .\yt-dlp.exe -vU https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/ --cookies-from-browser firefox
[debug] Command-line config: ['-vU', 'https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/', '--cookies-from-browser', 'firefox']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (win_exe)
[debug] Python 3.8.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1k  25 Mar 2021)
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[Cookies] Extracting cookies from firefox
[debug] Extracting cookies from: "C:\Users\XXXX\AppData\Roaming\Mozilla\Firefox\Profiles\9jedevhd.default-release-1667831011215\cookies.sqlite"
[Cookies] Extracted 2893 cookies from firefox
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
Current Build Hash: 5590c57bd0433ed239a2deaaf92e2ad6f37fe50f53664c821575cafe106a9421
yt-dlp is up to date (stable@2023.03.04)
[Reddit] Extracting URL: https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/
[Reddit] 12fujy3: Downloading JSON metadata
WARNING: [Reddit] Unable to download JSON metadata: HTTP Error 403: Forbidden
[Reddit] 12fujy3: Downloading JSON metadata
ERROR: [Reddit] 12fujy3: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp\extractor\common.py", line 694, in extract
  File "yt_dlp\extractor\reddit.py", line 154, in _real_extract
  File "yt_dlp\extractor\common.py", line 1048, in download_content
  File "yt_dlp\extractor\common.py", line 1012, in download_handle
  File "yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
  File "yt_dlp\extractor\common.py", line 839, in _request_webpage

  File "yt_dlp\extractor\common.py", line 821, in _request_webpage
  File "yt_dlp\YoutubeDL.py", line 3742, in urlopen
  File "urllib\request.py", line 531, in open
  File "urllib\request.py", line 640, in http_response
  File "urllib\request.py", line 569, in error
  File "urllib\request.py", line 502, in _call_chain
  File "urllib\request.py", line 649, in http_error_default
urllib.error.HTTPError: HTTP Error 403: Forbidden
```



Discussion:
The extractor's bypass method for accessing quarantined subreddits without an account no longer works. This broken workaround sets/overwrites the `reddit_session` cookie by default, rendering passed `--cookies` useless.

This will at least fix it so that passed cookies are effective:
```diff
diff --git a/yt_dlp/extractor/reddit.py b/yt_dlp/extractor/reddit.py
index 9dba3eca8..f42ae2887 100644
--- a/yt_dlp/extractor/reddit.py
+++ b/yt_dlp/extractor/reddit.py
@@ -146,7 +146,8 @@ def _gen_session_id():
     def _real_extract(self, url):
         subdomain, slug, video_id = self._match_valid_url(url).group('subdomain', 'slug', 'id')
 
-        self._set_cookie('.reddit.com', 'reddit_session', self._gen_session_id())
+        if not self._get_cookies('https://www.reddit.com/').get('reddit_session'):
+            self._set_cookie('.reddit.com', 'reddit_session', self._gen_session_id())
         self._set_cookie('.reddit.com', '_options', '%7B%22pref_quarantine_optin%22%3A%20true%7D')
         data = self._download_json(f'https://{subdomain}reddit.com/{slug}/.json', video_id, fatal=False)
         if not data:
```

with patch:
```
$ yt-dlp -F "https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/" --cookies-from-browser firefox:xu2iz3kn.default-release::dlp3
[Cookies] Extracting cookies from firefox
[Cookies] Extracted 17 cookies from firefox
[Reddit] Extracting URL: https://www.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/
[Reddit] 12fujy3: Downloading JSON metadata
[Reddit] 12fujy3: Downloading m3u8 information
[Reddit] 12fujy3: Downloading MPD manifest
[info] Available formats for 8bwtclfggpsa1:
ID                  EXT RESOLUTION FPS │   FILESIZE   TBR PROTO │ VCODEC        VBR ACODEC      ABR ASR MORE INFO
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
hls-3-audio_0       mp4 audio only     │                  m3u8  │ audio only        unknown             audio 0
hls-4-audio_0       mp4 audio only     │                  m3u8  │ audio only        unknown             audio 0
dash-audio_0_123864 m4a audio only     │ ~247.73KiB  124k https │ audio only        mp4a.40.2  124k 48k DASH audio, m4a_dash
dash-video_252009   mp4 390x220     30 │ ~504.02KiB  252k https │ avc1.4D401E  252k video only          DASH video, mp4_dash
hls-350             mp4 400x226        │ ~700.00KiB  350k m3u8  │ avc1.42c01e  350k video only
hls-438             mp4 400x226        │ ~876.00KiB  438k m3u8  │ avc1.42c01e  438k video only
dash-video_571171   mp4 426x240     30 │ ~  1.12MiB  571k https │ avc1.4D401E  571k video only          DASH video, mp4_dash
hls-551             mp4 480x270        │ ~  1.08MiB  551k m3u8  │ avc1.42c01e  551k video only
hls-639             mp4 480x270        │ ~  1.25MiB  639k m3u8  │ avc1.42c01e  639k video only
hls-936             mp4 640x360        │ ~  1.83MiB  936k m3u8  │ avc1.4d401e  936k video only
hls-1024            mp4 640x360        │ ~  2.00MiB 1024k m3u8  │ avc1.4d401e 1024k video only
dash-video_764488   mp4 640x360     30 │ ~  1.49MiB  764k https │ avc1.4D401E  764k video only          DASH video, mp4_dash
fallback            mp4 852x480        │ ~  2.34MiB 1200k https │ h264        1200k video only          DASH video, mp4_dash
dash-video_1137926  mp4 852x480     30 │ ~  2.22MiB 1138k https │ avc1.4D401F 1138k video only          DASH video, mp4_dash
```

not sure, but I think we could instead remove the bypass code altogether:
```diff
diff --git a/yt_dlp/extractor/reddit.py b/yt_dlp/extractor/reddit.py
index 9dba3eca8..29b9b3ebc 100644
--- a/yt_dlp/extractor/reddit.py
+++ b/yt_dlp/extractor/reddit.py
@@ -137,17 +137,8 @@ class RedditIE(InfoExtractor):
         'only_matching': True,
     }]
 
-    @staticmethod
-    def _gen_session_id():
-        id_length = 16
-        rand_max = 1 << (id_length * 4)
-        return '%0.*x' % (id_length, random.randrange(rand_max))
-
     def _real_extract(self, url):
         subdomain, slug, video_id = self._match_valid_url(url).group('subdomain', 'slug', 'id')
-
-        self._set_cookie('.reddit.com', 'reddit_session', self._gen_session_id())
-        self._set_cookie('.reddit.com', '_options', '%7B%22pref_quarantine_optin%22%3A%20true%7D')
         data = self._download_json(f'https://{subdomain}reddit.com/{slug}/.json', video_id, fatal=False)
         if not data:
             # Fall back to old.reddit.com in case the requested subdomain fails
```
just doing some quick testing I'm not experiencing any negative effects from the above patch. Haven't tested  with NSFW content though

a proper PR would ideally add error handling/messaging for 403 responses (there are reasons given in the response JSON)

Resolving patch:
diff --git a/yt_dlp/extractor/reddit.py b/yt_dlp/extractor/reddit.py
--- a/yt_dlp/extractor/reddit.py
+++ b/yt_dlp/extractor/reddit.py
@@ -1,4 +1,3 @@
-import random
 import urllib.parse
 
 from .common import InfoExtractor
@@ -14,7 +13,7 @@
 
 
 class RedditIE(InfoExtractor):
-    _VALID_URL = r'https?://(?P<subdomain>[^/]+\.)?reddit(?:media)?\.com/(?P<slug>(?:r|user)/[^/]+/comments/(?P<id>[^/?#&]+))'
+    _VALID_URL = r'https?://(?P<host>(?:\w+\.)?reddit(?:media)?\.com)/(?P<slug>(?:(?:r|user)/[^/]+/)?comments/(?P<id>[^/?#&]+))'
     _TESTS = [{
         'url': 'https://www.reddit.com/r/videos/comments/6rrwyj/that_small_heart_attack/',
         'info_dict': {
@@ -109,6 +108,46 @@ class RedditIE(InfoExtractor):
             'age_limit': 0,
             'channel_id': 'dumbfuckers_club',
         },
+    }, {
+        # post link without subreddit
+        'url': 'https://www.reddit.com/comments/124pp33',
+        'md5': '15eec9d828adcef4468b741a7e45a395',
+        'info_dict': {
+            'id': 'antsenjc2jqa1',
+            'ext': 'mp4',
+            'display_id': '124pp33',
+            'title': 'Harmless prank of some old friends',
+            'uploader': 'Dudezila',
+            'channel_id': 'ContagiousLaughter',
+            'duration': 17,
+            'upload_date': '20230328',
+            'timestamp': 1680012043,
+            'thumbnail': r're:^https?://.*\.(?:jpg|png)',
+            'age_limit': 0,
+            'comment_count': int,
+            'dislike_count': int,
+            'like_count': int,
+        },
+    }, {
+        # quarantined subreddit post
+        'url': 'https://old.reddit.com/r/GenZedong/comments/12fujy3/based_hasan/',
+        'md5': '3156ea69e3c1f1b6259683c5abd36e71',
+        'info_dict': {
+            'id': '8bwtclfggpsa1',
+            'ext': 'mp4',
+            'display_id': '12fujy3',
+            'title': 'Based Hasan?',
+            'uploader': 'KingNigelXLII',
+            'channel_id': 'GenZedong',
+            'duration': 16,
+            'upload_date': '20230408',
+            'timestamp': 1680979138,
+            'age_limit': 0,
+            'comment_count': int,
+            'dislike_count': int,
+            'like_count': int,
+        },
+        'skip': 'Requires account that has opted-in to the GenZedong subreddit',
     }, {
         'url': 'https://www.reddit.com/r/videos/comments/6rrwyj',
         'only_matching': True,
@@ -137,21 +176,26 @@ class RedditIE(InfoExtractor):
         'only_matching': True,
     }]
 
-    @staticmethod
-    def _gen_session_id():
-        id_length = 16
-        rand_max = 1 << (id_length * 4)
-        return '%0.*x' % (id_length, random.randrange(rand_max))
-
     def _real_extract(self, url):
-        subdomain, slug, video_id = self._match_valid_url(url).group('subdomain', 'slug', 'id')
+        host, slug, video_id = self._match_valid_url(url).group('host', 'slug', 'id')
 
-        self._set_cookie('.reddit.com', 'reddit_session', self._gen_session_id())
-        self._set_cookie('.reddit.com', '_options', '%7B%22pref_quarantine_optin%22%3A%20true%7D')
-        data = self._download_json(f'https://{subdomain}reddit.com/{slug}/.json', video_id, fatal=False)
+        data = self._download_json(
+            f'https://{host}/{slug}/.json', video_id, fatal=False, expected_status=403)
         if not data:
-            # Fall back to old.reddit.com in case the requested subdomain fails
-            data = self._download_json(f'https://old.reddit.com/{slug}/.json', video_id)
+            fallback_host = 'old.reddit.com' if host != 'old.reddit.com' else 'www.reddit.com'
+            self.to_screen(f'{host} request failed, retrying with {fallback_host}')
+            data = self._download_json(
+                f'https://{fallback_host}/{slug}/.json', video_id, expected_status=403)
+
+        if traverse_obj(data, 'error') == 403:
+            reason = data.get('reason')
+            if reason == 'quarantined':
+                self.raise_login_required('Quarantined subreddit; an account that has opted in is required')
+            elif reason == 'private':
+                self.raise_login_required('Private subreddit; an account that has been approved is required')
+            else:
+                raise ExtractorError(f'HTTP Error 403 Forbidden; reason given: {reason}')
+
         data = data[0]['data']['children'][0]['data']
         video_url = data['url']
 


Test patch: