Repository: yt-dlp/yt-dlp

Issue:
[generic] None: Unable to download webpage: HTTP Error 403: Forbidden
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

```bash
yt-dlp --extractor-args "generic:hls_key=https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.key?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5rZXk~c2Vzc2lvbklEPTVhNWE5MDdlLWUxYzItNGEyNC05ZWMzLTNlNzAxYTNjMzUwYyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3OTQ5NTg2OX19fV19&Signature=X9DwHbafzvP-4RrKTuYztmQtIFVXVon6v4-xLilhvk8Za1fQ-JLjAQoSgMPBrcryfX4WqsYmf4vfgc-zBi1vYcZO8vxcz-GIaAgzN8VlhYZYwb1v1-ZAWEFI-k6voCWGGF04c~rQmxlJJboG5y5dM0ZL6Oves5UVxTa3fm-NSibyNqXdJKto0ADAKd6UV6H3tHI-x87IOnoqaUvhx2JND5SYJUWUVqGmIW1cBMUkDrSo4l~gJrdNlNLJrVMK5P9e0rnjggJzqwO4pAM3pPgxbZNt0~0dMuWlEi2WfiZL5lZwZcFjTFrkogYOaBA-fFtXTlCD5njQngeiqJUg5hHKKA__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5a5a907e-e1c2-4a24-9ec3-3e701a3c350c" "https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb"
```

get this error: [generic] None: Unable to download webpage: HTTP Error 403: Forbidden

I write a simple demo for this url, it response 200

```python
import urllib.request

url = 'https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb'

req = urllib.request.Request(url)
response = urllib.request.urlopen(req)

print(response.code)
```
for the simple demo, it will respose 200.

the m3u8 url can get from this page: https://www.wanderlustentrepreneur.com/rysr-replays/

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[generic] Extracting URL: https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c9971...c6-91a1-30d0e95882eb 
kxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5rZXk~c2Vzc2lvbklEPTVhNWE5MDdlLWUxYzItNGEyNC05ZWMzLTNlNzAxYTNjMzUwYyIsIkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTY3OTQ5NTg2OX19fV19&Signature=X9DwHbafzvP-4RrKTuYztmQtIFVXVon6v4-xLilhvk8Za1fQ-JLjAQoSgMPBrcryfX4WqsYmf4vfgc-zBi1vYcZO8vxcz-GIaAgzN8VlhYZYwb1v1-ZAWEFI-k6voCWGGF04c~rQmxlJJboG5y5dM0ZL6Oves5UVxTa3fm-NSibyNqXdJKto0ADAKd6UV6H3tHI-x87IOnoqaUvhx2JND5SYJUWUVqGmIW1cBMUkDrSo4l~gJrdNlNLJrVMK5P9e0rnjggJzqwO4pAM3pPgxbZNt0~0dMuWlEi2WfiZL5lZwZcFjTFrkogYOaBA-fFtXTlCD5njQngeiqJUg5hHKKA__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5a5a907e-e1c2-4a24-9ec3-3e701a3c350c' 'https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240.m3u8?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi5tM3U4P3Nlc3Npb25JRD01ZjE4ZDY1ZS0yNzRjLTQ3YzYtOTFhMS0zMGQwZTk1ODgyZWIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE2Nzk0OTc3MDN9fX1dfQ__&Signature=ZbWNlPzMD9oAk7XmG6MrmLnNN7OwpzKYMGLj~2L9LTXafsaq5DHALky5lcJ7-PtmkfBnKleIvuA-DKDWRNCeAtLZULoYV-WQAlNMiv1qKv5hpbpbz5qekJTygl7d-C6G-7Y7QF1SxueGnKHy9AO8QCzOVHLyo78od0LSa0RDUQmfxc1OXny-Waj28C~d8iOStY8-ygYZ8GAg1kLxjjuoolMltm8CisAi07xzJQUY-0O7~bpZHyPFrudn4Tg-QvNQunXFsMEQFbLz7Nq-9Bjw0jz89r58fHbHoqL96-UkAbCXijXdTHgPLH7ndThhK2LvhH9aZ5XclskQVi9wrHeC3Q__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=5f18d65e-274c-47c6-91a1-30d0e95882eb'[generic] Extracting URL: https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c9971...c6-91a1-30d0e95882eb[generic] 240: Downloading webpageERROR: [generic] None: Unable to download webpage: HTTP Error 403: Forbidden (caused by <HTTPError 403: 'Forbidden'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```



Discussion:
Seems that the server is blocking based on user-agent. Adding this allows me to download the m3u8:
```
--add-header "User-Agent:Mozilla/5.0 (Windows NT 10.0; rv:111.0) Gecko/20100101 Firefox/111.0"
```

But then the fragments give 403 errors, even with the `fragment_query` extractor-arg.

The fragments 403 in browser too, with a message of `Missing Key-Pair-Id query parameter or cookie value` if I don't add the query string, and simply `AccessDenied` if I do add the query string.

(`hls_key` extractor-arg is irrelevant since yt-dlp never gets to the point of attempting to download the key)
I use fragment_query for it, it still get 403 error for the fragments

```bash
"--extractor-args" "generic:fragment_query=https://hls2.videos.sproutvideo.com/a3f063147006cc49918f18f5f40e209e/ac7f47322156a22444b8957c99719a62/video/240_00000.ts?Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cHM6Ly9obHMyLnZpZGVvcy5zcHJvdXR2aWRlby5jb20vYTNmMDYzMTQ3MDA2Y2M0OTkxOGYxOGY1ZjQwZTIwOWUvYWM3ZjQ3MzIyMTU2YTIyNDQ0Yjg5NTdjOTk3MTlhNjIvKi50cz9zZXNzaW9uSUQ9MzcwYzc3NzUtMTg0MC00ZmFiLTg2OGMtOTE1MjM2OWI1ZGFlIiwiQ29uZGl0aW9uIjp7IkRhdGVMZXNzVGhhbiI6eyJBV1M6RXBvY2hUaW1lIjoxNjc5NTA1Mzk0fX19XX0_&Signature=rKaHTLmmhdYdkJwcfRpT-6QsoK-Ldeo5wrZhANLoGuNGTtAmPCXAe1V2w~11p8XgJqZxuA5uVtP0qNyCtvULjjBx1l2IdWaGgWT3ROSvWClrsPCG33Qnl3EBQpZ0pxeUZdWUL42SvqxHlBq3ar3vCzP4V04OVo6ZqZGfJaKBG6GNP1wE~AsbToOh~zzaaLL9W78fkgE3U6odR0lzHxZSzgQXsxWyUJJ1iVpRPCsS~niT-0Hrt8b3gPMqy9Bom-4jhWm7x5GdGDLCEmOapmGM5ABa8QYLeuv7XHv23ianxYKBjyuaRQcpTWt~dnHsvM8xnySCJLK-WXJt-faylj5vdQ__&Key-Pair-Id=APKAIB5DGCGAQJ4GGIUQ&sessionID=370c7775-1840-4fab-868c-9152369b5dae"
```
`fragment_query` currently doesn't take an argument, it's a boolean-like switch that only passes through the m3u8 URL's query to the fragment URLs

Though maybe it should be improved so that it can take an optional argument
Repro with yt-dl, Which punts the HLS to _ffmpeg_ for the download and fails in a similar way. The manifest can't be played by _mpv_. A trace of the fragments being downloaded might help.

The containing page has a slightly interesting feature that could be a target if it didn't have a JS-enabled download link: the embed `<iframe>` has an element `<script>var dat = '{base64_encoded_js_object}';</script>`. `dat` is passed as an argument to the player.js. The leading encoded text `eyJ` in the `dat` value indicates `{"` followed by an alphabetic character.

Just `Mozilla/5.0` as UA gets the manifest itself.

Resolving patch:
diff --git a/yt_dlp/extractor/generic.py b/yt_dlp/extractor/generic.py
--- a/yt_dlp/extractor/generic.py
+++ b/yt_dlp/extractor/generic.py
@@ -24,7 +24,6 @@
     mimetype2ext,
     orderedSet,
     parse_duration,
-    parse_qs,
     parse_resolution,
     smuggle_url,
     str_or_none,
@@ -2187,18 +2186,23 @@ def report_detected(self, name, num=1, note=None):
         self._downloader.write_debug(f'Identified {num} {name}{format_field(note, None, "; %s")}')
 
     def _extra_manifest_info(self, info, manifest_url):
-        if self._configuration_arg('fragment_query'):
-            query_string = urllib.parse.urlparse(manifest_url).query
-            if query_string:
-                info['extra_param_to_segment_url'] = query_string
+        fragment_query = self._configuration_arg('fragment_query', [None], casesense=True)[0]
+        if fragment_query is not None:
+            fragment_query = self._configuration_arg('fragment_query', casesense=True)[0]
+            info['extra_param_to_segment_url'] = (
+                urllib.parse.urlparse(fragment_query).query or fragment_query
+                or urllib.parse.urlparse(manifest_url).query or None)
 
         hex_or_none = lambda x: x if re.fullmatch(r'(0x)?[\da-f]+', x, re.IGNORECASE) else None
-        info['hls_aes'] = traverse_obj(self._configuration_arg('hls_key'), {
+        info['hls_aes'] = traverse_obj(self._configuration_arg('hls_key', casesense=True), {
             'uri': (0, {url_or_none}), 'key': (0, {hex_or_none}), 'iv': (1, {hex_or_none}),
         }) or None
 
-        if self._configuration_arg('variant_query'):
-            query = parse_qs(manifest_url)
+        variant_query = self._configuration_arg('variant_query', [None], casesense=True)[0]
+        if variant_query is not None:
+            query = urllib.parse.parse_qs(
+                urllib.parse.urlparse(variant_query).query or variant_query
+                or urllib.parse.urlparse(manifest_url).query)
             for fmt in self._downloader._get_formats(info):
                 fmt['url'] = update_url_query(fmt['url'], query)
 


Test patch: