Repository: yt-dlp/yt-dlp

Issue:
YouTube subtitle downloads are unnecessarily broken when using a custom user agent string
### Checklist

- [X] I'm reporting a broken site
- [X] I've verified that I'm running yt-dlp version **2022.02.04**. ([update instructions](https://github.com/yt-dlp/yt-dlp#update))
- [X] I've checked that all provided URLs are alive and playable in a browser
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/ytdl-org/youtube-dl#video-url-contains-an-ampersand-and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues including closed ones. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

The included specific example content URL should be globally accessible, tested by me from Sweden though

### Description

Because of #2716 I've been using a custom user agent string with yt-dlp so much recently that I included it in my standard configuration to see if it would cause any trouble.
Now I've seen that including it somehow breaks subtitle downloads with YouTube content.

From the log output provided below it seems that the problem is caused by the custom user agent string somehow making YouTube or the extractor to behave differently leading to the subtitle downloader being invoked improperly; the leading "https://www.youtube.com" of the subtitle URL is missing.

Expected behavior would be that subtitle downloads worked fine even with the custom user agent string, especially since it seems the subtitle URL is supposed to be the same but somehow malformed on the way.


### Verbose log

```shell
Video download with the custom user agent string but without subs download works fine:

Z:\temp>yt-dlp --ignore-config --force-overwrites --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/36.0  Mobile/15E148 Safari/605.1.15" https://www.youtube.com/watch?v=fb5ELWi-ekk
[youtube] fb5ELWi-ekk: Downloading webpage
[youtube] fb5ELWi-ekk: Downloading android player API JSON
[youtube] fb5ELWi-ekk: Downloading initial data API JSON
[info] fb5ELWi-ekk: Downloading 1 format(s): 248+251
Deleting existing file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm
[download] 100% of 37.26MiB in 00:02
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm
[download] 100% of 2.60MiB in 00:00
[Merger] Merging formats into "Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm"
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm (pass -k to keep)
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm (pass -k to keep)
```

Video download with subs download but without the custom user agent string works fine:
```
Z:\temp>yt-dlp --ignore-config --force-overwrites --write-subs https://www.youtube.com/watch?v=fb5ELWi-ekk
[youtube] fb5ELWi-ekk: Downloading webpage
[youtube] fb5ELWi-ekk: Downloading android player API JSON
[info] fb5ELWi-ekk: Downloading 1 format(s): 248+251
[info] Writing video subtitles to: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[download] 100% of 1.54KiB in 00:00
Deleting existing file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm
[download] 100% of 37.26MiB in 00:02
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm
[download] 100% of 2.60MiB in 00:00
[Merger] Merging formats into "Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm"
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm (pass -k to keep)
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm (pass -k to keep)
```

Video download combining the custom user agent string with subs download doesn't work (but note the almost correct URL in the error):
```
Z:\temp>yt-dlp --ignore-config --force-overwrites --write-subs --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/36.0  Mobile/15E148 Safari/605.1.15" https://www.youtube.com/watch?v=fb5ELWi-ekk
[youtube] fb5ELWi-ekk: Downloading webpage
[youtube] fb5ELWi-ekk: Downloading android player API JSON
[youtube] fb5ELWi-ekk: Downloading initial data API JSON
[info] fb5ELWi-ekk: Downloading 1 format(s): 248+251
[info] Writing video subtitles to: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
ERROR: Unable to download video subtitles for 'en': unknown url type: '/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644775735&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=10191132E82E56B6A539F5F9F2B856FFC909A3B3.25365C17734D0D1F2A73F5341CEE0D483C305B9D&key=yt8&lang=en&fmt=vtt'
```

Same as the last one but with -vU:
```
Z:\temp>yt-dlp -vU --ignore-config --force-overwrites --write-subs --user-agent "Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/36.0  Mobile/15E148 Safari/605.1.15" https://www.youtube.com/watch?v=fb5ELWi-ekk
[debug] Command-line config: ['-vU', '--ignore-config', '--force-overwrites', '--write-subs', '--user-agent', 'Mozilla/5.0 (iPhone; CPU iPhone OS 14_7_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/36.0  Mobile/15E148 Safari/605.1.15', 'https://www.youtube.com/watch?v=fb5ELWi-ekk']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.02.04 [c1653e9] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev (setts), ffprobe 4.4-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=fb5ELWi-ekk
[youtube] fb5ELWi-ekk: Downloading webpage
[youtube] fb5ELWi-ekk: Downloading android player API JSON
[debug] Loading youtube-nsig.96dcbc8c from cache
[debug] [youtube] Decrypted nsig hoTtb9ObH1G9OfG => iMfwub6WNGV8vg
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[youtube] fb5ELWi-ekk: Downloading initial data API JSON
[debug] Downloading subtitles: en
[debug] Default format spec: bestvideo*+bestaudio/best
[info] fb5ELWi-ekk: Downloading 1 format(s): 248+251
Deleting existing file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[info] Writing video subtitles to: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[debug] Invoking downloader on "/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644775916&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=672EB2E52C220D63E06E56115D7CDF3502AA0CC0.6E07145938F269F41FD81C572814AFE1CDA5C8E1&key=yt8&lang=en&fmt=vtt"
ERROR: Unable to download video subtitles for 'en': unknown url type: '/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644775916&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=672EB2E52C220D63E06E56115D7CDF3502AA0CC0.6E07145938F269F41FD81C572814AFE1CDA5C8E1&key=yt8&lang=en&fmt=vtt'
Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 3825, in _write_subtitles
  File "yt_dlp\YoutubeDL.py", line 2800, in dl
  File "yt_dlp\YoutubeDL.py", line 2245, in _calc_headers
  File "yt_dlp\YoutubeDL.py", line 2257, in _calc_cookies
  File "yt_dlp\utils.py", line 798, in sanitized_Request
  File "urllib\request.py", line 328, in __init__
  File "urllib\request.py", line 354, in full_url
  File "urllib\request.py", line 383, in _parse
ValueError: unknown url type: '/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644775916&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=672EB2E52C220D63E06E56115D7CDF3502AA0CC0.6E07145938F269F41FD81C572814AFE1CDA5C8E1&key=yt8&lang=en&fmt=vtt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "yt_dlp\YoutubeDL.py", line 1381, in wrapper
  File "yt_dlp\YoutubeDL.py", line 1465, in __extract_info
  File "yt_dlp\YoutubeDL.py", line 1517, in process_ie_result
  File "yt_dlp\YoutubeDL.py", line 2607, in process_video_result
  File "yt_dlp\YoutubeDL.py", line 2852, in process_info
  File "yt_dlp\YoutubeDL.py", line 3830, in _write_subtitles
yt_dlp.utils.DownloadError: Unable to download video subtitles for 'en': unknown url type: '/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644775916&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=672EB2E52C220D63E06E56115D7CDF3502AA0CC0.6E07145938F269F41FD81C572814AFE1CDA5C8E1&key=yt8&lang=en&fmt=vtt'
```

For comparison, with subs download but without custom user agent string again, now with -vU (notice that the subtitle downloader is invoked with a slightly different URL now):
```
Z:\temp>yt-dlp -vU --ignore-config --force-overwrites --write-subs https://www.youtube.com/watch?v=fb5ELWi-ekk
[debug] Command-line config: ['-vU', '--ignore-config', '--force-overwrites', '--write-subs', 'https://www.youtube.com/watch?v=fb5ELWi-ekk']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, err utf-8, pref cp1252
[debug] yt-dlp version 2022.02.04 [c1653e9] (win_exe)
[debug] Python version 3.8.10 (CPython 64bit) - Windows-10-10.0.17763-SP0
[debug] exe versions: ffmpeg 4.4-full_build-www.gyan.dev (setts), ffprobe 4.4-full_build-www.gyan.dev
[debug] Optional libraries: Cryptodome, mutagen, sqlite, websockets
[debug] Proxy map: {}
Latest version: 2022.02.04, Current version: 2022.02.04
yt-dlp is up to date (2022.02.04)
[debug] [youtube] Extracting URL: https://www.youtube.com/watch?v=fb5ELWi-ekk
[youtube] fb5ELWi-ekk: Downloading webpage
[youtube] fb5ELWi-ekk: Downloading android player API JSON
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, codec:vp9.2, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[debug] Downloading subtitles: en
[debug] Default format spec: bestvideo*+bestaudio/best
[info] fb5ELWi-ekk: Downloading 1 format(s): 248+251
[info] Writing video subtitles to: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[debug] Invoking downloader on "https://www.youtube.com/api/timedtext?v=fb5ELWi-ekk&exp=xctw&xoaf=5&hl=en&ip=0.0.0.0&ipbits=0&expire=1644776226&sparams=ip%2Cipbits%2Cexpire%2Cv%2Cexp%2Cxoaf&signature=7A4C763303C4A0415D577A8B8315BEFB01E88DF2.95ABE541F8E6E2F34FDC8F3363FAEB710D692497&key=yt8&lang=en&fmt=vtt"
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].en.vtt
[download] 100% of 1.54KiB in 00:00
Deleting existing file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm
[debug] Invoking downloader on "https://rr6---sn-jvhixh-5go6.googlevideo.com/videoplayback?expire=1644772626&ei=sugIYo2MIoHRyQW-sJ-4CA&ip=188.151.16.231&id=o-AEN_PSjMkIITIKprE2R9wd0sOII2NvGAuIpRpp-U-OZ2&itag=248&source=youtube&requiressl=yes&mh=HN&mm=31%2C26&mn=sn-jvhixh-5go6%2Csn-5hne6nzd&ms=au%2Conr&mv=m&mvi=6&pcm2cms=yes&pl=25&initcwndbps=2702500&vprv=1&mime=video%2Fwebm&gir=yes&clen=39072182&dur=171.046&lmt=1644525618928180&mt=1644750553&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5535434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRAIgDcytN0PHA4m5rvYRLsbpsZFDB_VEs6DiaF0rQkiRYzwCIAFaVL2t9uTN4YM54-QN5kJqDqS1tIGe2RBNI3iP2-On&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAK4yftC66vdrXHgnL6TkNHX0ijnSvE28g7ShbUffMsteAiEA3XMk2y00cLLoflo3_3VQNGtVumaC88rFHVWd1RYe9Fk%3D"
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm
[download] 100% of 37.26MiB in 00:02
[debug] Invoking downloader on "https://rr6---sn-jvhixh-5go6.googlevideo.com/videoplayback?expire=1644772626&ei=sugIYo2MIoHRyQW-sJ-4CA&ip=188.151.16.231&id=o-AEN_PSjMkIITIKprE2R9wd0sOII2NvGAuIpRpp-U-OZ2&itag=251&source=youtube&requiressl=yes&mh=HN&mm=31%2C26&mn=sn-jvhixh-5go6%2Csn-5hne6nzd&ms=au%2Conr&mv=m&mvi=6&pcm2cms=yes&pl=25&initcwndbps=2702500&vprv=1&mime=audio%2Fwebm&gir=yes&clen=2731354&dur=171.061&lmt=1644520981416726&mt=1644750553&fvip=5&keepalive=yes&fexp=24001373%2C24007246&c=ANDROID&txp=5532434&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cgir%2Cclen%2Cdur%2Clmt&sig=AOq0QJ8wRQIgMgu-TXmng1sFLEDIEZNOsqUqjfpV7CXdR10BrhoRf98CIQDGLdG_HR-xRxvjKejgHvGeiChn1_1QJsmKhtihrUbzAw%3D%3D&lsparams=mh%2Cmm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpcm2cms%2Cpl%2Cinitcwndbps&lsig=AG3C_xAwRgIhAK4yftC66vdrXHgnL6TkNHX0ijnSvE28g7ShbUffMsteAiEA3XMk2y00cLLoflo3_3VQNGtVumaC88rFHVWd1RYe9Fk%3D"
[download] Destination: Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm
[download] 100% of 2.60MiB in 00:00
[Merger] Merging formats into "Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].webm"
[debug] ffmpeg command line: ffmpeg -y -loglevel "repeat+info" -i "file:Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm" -i "file:Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm" -c copy -map "0:v:0" -map "1:a:0" -movflags "+faststart" "file:Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].temp.webm"
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f248.webm (pass -k to keep)
Deleting original file Jurassic World Dominion - Official Trailer [HD] [fb5ELWi-ekk].f251.webm (pass -k to keep)
```
```



Discussion:
Youtube may server different data depending on the UA used. That is the whole point of using the `--user-agent` option! I haven't tested that is indeed the issue here, but if it is, this is expected behavior and there is nothing to be done about it
This UA appears to make the webpage load the mweb client, and thus the player response is too. mweb seems to give relative urls for the subtitles.

Indeed, I can reproduce with `yt-dlp --extractor-args "youtube:player-client=mweb;player-skip=webpage" --write-subs fb5ELWi-ekk`

In fact, this occurs when using multiple clients with mweb being one of them:

`yt-dlp --extractor-args "youtube:player-client=default,mweb" --write-subs fb5ELWi-ekk`

Fix would be to ensure the relative URL is absolute, similar to what we do for the player js url.




Resolving patch:
diff --git a/yt_dlp/extractor/youtube.py b/yt_dlp/extractor/youtube.py
--- a/yt_dlp/extractor/youtube.py
+++ b/yt_dlp/extractor/youtube.py
@@ -2245,12 +2245,7 @@ def _extract_player_url(self, *ytcfgs, webpage=None):
             get_all=False, expected_type=compat_str)
         if not player_url:
             return
-        if player_url.startswith('//'):
-            player_url = 'https:' + player_url
-        elif not re.match(r'https?://', player_url):
-            player_url = compat_urlparse.urljoin(
-                'https://www.youtube.com', player_url)
-        return player_url
+        return urljoin('https://www.youtube.com', player_url)
 
     def _download_player_url(self, video_id, fatal=False):
         res = self._download_webpage(
@@ -2399,11 +2394,7 @@ def _decrypt_nsig(self, s, video_id, player_url):
         """Turn the encrypted n field into a working signature"""
         if player_url is None:
             raise ExtractorError('Cannot decrypt nsig without player_url')
-        if player_url.startswith('//'):
-            player_url = 'https:' + player_url
-        elif not re.match(r'https?://', player_url):
-            player_url = compat_urlparse.urljoin(
-                'https://www.youtube.com', player_url)
+        player_url = urljoin('https://www.youtube.com', player_url)
 
         sig_id = ('nsig_value', s)
         if sig_id in self._player_cache:
@@ -3383,7 +3374,7 @@ def process_language(container, base_url, lang_code, sub_name, query):
                     })
                     lang_subs.append({
                         'ext': fmt,
-                        'url': update_url_query(base_url, query),
+                        'url': urljoin('https://www.youtube.com', update_url_query(base_url, query)),
                         'name': sub_name,
                     })
 


Test patch: