Repository: yt-dlp/yt-dlp

Issue:
[NoodleMagazine] Unable to download JSON metadata: HTTP Error 403: Forbidden
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I'm running yt-dlp version **2023.07.06** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

UK

### Provide a description that is worded well enough to be understood

Getting the 403 forbidden error when downloading from `noodlemagazine.com`. Works fine from browsers. Login is not necessary. Tried with vpn, used `--user-agent` and `--cookies` parameters, updated yt_dlp using master branch, but all to no avail.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['-vU', '--user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36', '--cookies', '.\\cookies.txt', 'https://noodlemagazine.com/watch/-221537140_456239050']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.07.06 [b532a3481]
[debug] Lazy loading extractors is disabled
[debug] Python 3.11.4 (CPython AMD64 64bit) - Windows-10-10.0.19045-SP0 (OpenSSL 1.1.1u  30 May 2023)
[debug] exe versions: ffmpeg 2023-07-19-git-efa6cec759-full_build-www.gyan.dev (setts), ffprobe 2023-07-19-git-efa6cec759-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.07.22, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1863 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.07.06, Current version: stable@2023.07.06
yt-dlp is up to date (stable@2023.07.06)
[NoodleMagazine] Extracting URL: https://noodlemagazine.com/watch/-221537140_456239050
[NoodleMagazine] -221537140_456239050: Downloading webpage
[NoodleMagazine] -221537140_456239050: Downloading JSON metadata
ERROR: [NoodleMagazine] -221537140_456239050: Unable to download JSON metadata: HTTP Error 403: Forbidden (caused by <HTTPError 403: Forbidden>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 715, in extract
    ie_result = self._real_extract(url)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\noodlemagazine.py", line 41, in _real_extract
    playlist_info = self._download_json(f'https://adult.noodlemagazine.com/playlist/{video_id}?m={key}', video_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 1069, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 1033, in download_handle
    res = self._download_webpage_handle(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 903, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 860, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_urllib.py", line 438, in _send
    res = opener.open(urllib_req, timeout=float(request.extensions.get('timeout') or self.timeout))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 525, in open
    response = meth(req, response)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
               ^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
             ^^^^^^^^^^^
  File "C:\Users\****\AppData\Local\Programs\Python\Python311\Lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\YoutubeDL.py", line 4059, in urlopen
    return self._request_director.send(req)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\common.py", line 114, in send
    response = handler.send(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_helper.py", line 203, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\common.py", line 325, in send
    return self._send(request)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\networking\_urllib.py", line 443, in _send
    raise HTTPError(UrllibResponseAdapter(e.fp), redirect_loop='redirect error' in str(e)) from e
yt_dlp.networking.exceptions.HTTPError: HTTP Error 403: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\extractor\common.py", line 847, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\****\Downloads\venv_yt_dlp\Lib\site-packages\yt_dlp\YoutubeDL.py", line 4078, in urlopen
    raise _CompatHTTPError(e) from e
yt_dlp.networking.exceptions._CompatHTTPError: HTTP Error 403: Forbidden
```



Discussion:

Resolving patch:
diff --git a/yt_dlp/extractor/noodlemagazine.py b/yt_dlp/extractor/noodlemagazine.py
--- a/yt_dlp/extractor/noodlemagazine.py
+++ b/yt_dlp/extractor/noodlemagazine.py
@@ -1,9 +1,14 @@
 from .common import InfoExtractor
 from ..utils import (
-    parse_duration,
+    extract_attributes,
+    get_element_html_by_id,
+    int_or_none,
     parse_count,
-    unified_strdate
+    parse_duration,
+    unified_strdate,
+    urljoin,
 )
+from ..utils.traversal import traverse_obj
 
 
 class NoodleMagazineIE(InfoExtractor):
@@ -37,15 +42,21 @@ def _real_extract(self, url):
         like_count = parse_count(self._html_search_meta('ya:ovs:likes', webpage, default=None))
         upload_date = unified_strdate(self._html_search_meta('ya:ovs:upload_date', webpage, default=''))
 
-        key = self._html_search_regex(rf'/{video_id}\?(?:.*&)?m=([^&"\'\s,]+)', webpage, 'key')
-        playlist_info = self._download_json(f'https://adult.noodlemagazine.com/playlist/{video_id}?m={key}', video_id)
-        thumbnail = self._og_search_property('image', webpage, default=None) or playlist_info.get('image')
+        player_path = extract_attributes(get_element_html_by_id('iplayer', webpage) or '')['src']
+        player_iframe = self._download_webpage(
+            urljoin('https://adult.noodlemagazine.com', player_path), video_id, 'Downloading iframe page')
+        playlist_url = self._search_regex(
+            r'window\.playlistUrl\s*=\s*["\']([^"\']+)["\']', player_iframe, 'playlist url')
+        playlist_info = self._download_json(
+            urljoin('https://adult.noodlemagazine.com', playlist_url), video_id, headers={'Referer': url})
 
-        formats = [{
-            'url': source.get('file'),
-            'quality': source.get('label'),
-            'ext': source.get('type'),
-        } for source in playlist_info.get('sources')]
+        thumbnail = self._og_search_property('image', webpage, default=None) or playlist_info.get('image')
+        formats = traverse_obj(playlist_info, ('sources', lambda _, v: v['file'], {
+            'url': 'file',
+            'format_id': 'label',
+            'height': ('label', {int_or_none}),
+            'ext': 'type',
+        }))
 
         return {
             'id': video_id,


Test patch: