Repository: yt-dlp/yt-dlp

Issue:
HTTPError 426: 'Upgrade Required' when downloading from CBC
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting a bug unrelated to a specific site
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)

### Provide a description that is worded well enough to be understood

Hello! I was downloading from CBC just fine for a few days and everything was fine. Suddenly all my CBC downloads started failing with the following error:
```
yt-dlp -o "C:\Users\...\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```

> yt-dlp -U
```
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
```

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [X] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
yt_dlp -vU -o "C:\<redacted>\yt-dlp\%%(channel)s - %%(title)s.%%(ext)s" https://gem.cbc.ca/media/the-fifth-estate/s41e09
[debug] Command-line config: ['-vU', '-o', 'C:\\<redacted>\\yt-dlp\\%%(channel)s - %%(title)s.%%(ext)s', 'https://gem.cbc.ca/media/the-fifth-estate/s41e09']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 68be95bd0
[debug] Python 3.10.10 (CPython AMD64 64bit) - Windows-10-10.0.22621-SP0 (OpenSSL 1.1.1s  1 Nov 2022)
[debug] exe versions: none
[debug] Optional libraries: sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[debug] Using fake IP 99.230.144.152 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/the-fifth-estate/s41e09
[gem.cbc.ca] the-fifth-estate/s41e09: Downloading JSON metadata
ERROR: [gem.cbc.ca] the-fifth-estate/s41e09: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "C:\<redacted>\yt_dlp\extractor\cbc.py", line 352, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "C:\<redacted>\yt_dlp\extractor\common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "C:\<redacted>\yt_dlp\extractor\common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "C:\<redacted>\YoutubeDL.py", line 3743, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 525, in open
    response = meth(req, response)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 634, in http_response
    response = self.parent.error(
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 563, in error
    return self._call_chain(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 496, in _call_chain
    result = func(*args)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.2800.0_x64__qbz5n2kfra8p0\lib\urllib\request.py", line 643, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required
```



Discussion:
Is this a bug with the yt-dlp extractor? Downloading 720p works (directly) but -F doesn't show the 1080p stream.
I can confirm this issue, I will look into it. Worst case I will be free to explore this in a couple weeks.
@makew0rld thank you, I for one will appreciate it.
I also have this issue, the http traffic shows an interesting Upgrade header:

```[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/murdoch-mysteries/s11e08
[gem.cbc.ca] murdoch-mysteries/s11e08: Downloading JSON metadata
send: b'GET /ott/cbc-api/v2/assets/murdoch-mysteries/s11e08 HTTP/1.1\r\nHost: services.radio-canada.ca\r\nX-Forwarded-For: 99.247.151.173\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.74 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nAccept-Language: en-us,en;q=0.5\r\nSec-Fetch-Mode: navigate\r\nAccept-Encoding: gzip, deflate, br\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 426 Upgrade Required\r\n'
header: Server: nginx/1.23.2
header: Content-Type: application/json
header: Content-Length: 2398
header: Expires: Fri, 07 Apr 2023 19:05:47 GMT
header: Cache-Control: max-age=0, no-cache, no-store
header: Pragma: no-cache
header: Date: Fri, 07 Apr 2023 19:05:47 GMT
header: Connection: close
header: Upgrade: 11.0.0
header: gem-api-version: 1.20.0
ERROR: [gem.cbc.ca] murdoch-mysteries/s11e08: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```
So we need to upgrade the protocol sent from yt-dlp.
I will investigate what the browser client does, as that is what is best to copy. But when I actually visit the [URL](https://services.radio-canada.ca/ott/cbc-api/v2/assets/murdoch-mysteries/s11e08), while it does return HTTP code 426 and have that non-standard Upgrade header, all the expected JSON metadata is there. I suspect if youtube-dl just ignored the HTTP code things would continue working fine.

Will look more into this later as mentioned above.
@makew0rld I wanted to test your theory and was able to confirm that you are correct.

As a test I added `expected_status = 426` here: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/common.py#L881, recompiled my executable and confirmed it worked.

Before:
```
redacted@redacted:/mnt/c/Users/redacted/Downloads$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1786 extractors
[debug] Using fake IP 99.229.117.88 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
ERROR: [gem.cbc.ca] plan-b/s01e03: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "./yt-dlp/yt_dlp/extractor/common.py", line 694, in extract
    ie_result = self._real_extract(url)
  File "./yt-dlp/yt_dlp/extractor/cbc.py", line 349, in _real_extract
    video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1048, in download_content
    res = getattr(self, download_handle.__name__)(url_or_request, video_id, **kwargs)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 1012, in download_handle
    res = self._download_webpage_handle(
  File "./yt-dlp/yt_dlp/extractor/common.py", line 882, in _download_webpage_handle
    urlh = self._request_webpage(url_or_request, video_id, note, errnote, fatal, data=data, headers=headers, query=query, expected_status=expected_status)
  File "./yt-dlp/yt_dlp/extractor/common.py", line 839, in _request_webpage
    raise ExtractorError(errmsg, cause=err)

  File "./yt-dlp/yt_dlp/extractor/common.py", line 821, in _request_webpage
    return self._downloader.urlopen(self._create_request(url_or_request, data, headers, query))
  File "./yt-dlp/yt_dlp/YoutubeDL.py", line 3742, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 426: Upgrade Required
```

After:
```
redacted@redacted:/mnt/d/Code/yt-dlp$ ./yt-dlp https://gem.cbc.ca/media/plan-b/s01e03 -v
[debug] Command-line config: ['https://gem.cbc.ca/media/plan-b/s01e03', '-v']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (zip)
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: Cryptodome-3.17, brotli-1.0.9, certifi-2022.12.07, mutagen-1.46.0, secretstorage-2.3.1, sqlite3-2.6.0, websockets-11.0.1
[debug] Proxy map: {}
[debug] Loaded 1791 extractors
[debug] Using fake IP 99.235.152.163 (CA) as X-Forwarded-For
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/plan-b/s01e03
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading JSON metadata
[gem.cbc.ca] plan-b/s01e03: Downloading m3u8 information
[gem.cbc.ca] plan-b/s01e03: Downloading secret XML
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[debug] Default format spec: bestvideo*+bestaudio/best
[info] plan-b/s01e03: Downloading 1 format(s): sec-1080+hls-audio-English
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(5999851)/Manifest(video,format=m3u8-aapl)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 438
[download] Destination: Episode 3 [plan-b⧸s01e03].fsec-1080.mp4
[download] 100% of    1.88GiB in 00:01:58 at 16.24MiB/s
[debug] Invoking hlsnative downloader on "https://cbcrcott-gem.akamaized.net/9a7e7072-ad0a-4e25-b860-2a9f40c8761a/CBC_PLANB_CBC_S01E03.ism/QualityLevels(192000)/Manifest(audio_eng_aacl,format=m3u8-aapl,filter=desktop)"
[hlsnative] Downloading m3u8 manifest
[hlsnative] Total fragments: 435
[download] Destination: Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a
[download] 100% of   62.86MiB in 00:00:39 at 1.57MiB/s
[debug] ffmpeg command line: ffprobe -show_streams 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a'
[Merger] Merging formats into "Episode 3 [plan-b⧸s01e03].mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i 'file:Episode 3 [plan-b⧸s01e03].fsec-1080.mp4' -i 'file:Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a' -c copy -map 0:v:0 -map 1:a:0 -bsf:a:0 aac_adtstoasc -movflags +faststart 'file:Episode 3 [plan-b⧸s01e03].temp.mp4'
Deleting original file Episode 3 [plan-b⧸s01e03].fhls-audio-English.m4a (pass -k to keep)
Deleting original file Episode 3 [plan-b⧸s01e03].fsec-1080.mp4 (pass -k to keep)
```
My above message wasn't necessarily a suggestion for a patch, was just a dirty confirmation of the theory.

This would be more gracious:

This line can be modified: https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/cbc.py#L352

from:
`video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)`

to 
`video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id, expected_status=426)`

Resolving patch:
diff --git a/yt_dlp/extractor/cbc.py b/yt_dlp/extractor/cbc.py
--- a/yt_dlp/extractor/cbc.py
+++ b/yt_dlp/extractor/cbc.py
@@ -349,7 +349,9 @@ def _find_secret_formats(self, formats, video_id):
 
     def _real_extract(self, url):
         video_id = self._match_id(url)
-        video_info = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/assets/' + video_id, video_id)
+        video_info = self._download_json(
+            f'https://services.radio-canada.ca/ott/cbc-api/v2/assets/{video_id}',
+            video_id, expected_status=426)
 
         email, password = self._get_login_info()
         if email and password:
@@ -421,7 +423,7 @@ def _real_extract(self, url):
         match = self._match_valid_url(url)
         season_id = match.group('id')
         show = match.group('show')
-        show_info = self._download_json(self._API_BASE + show, season_id)
+        show_info = self._download_json(self._API_BASE + show, season_id, expected_status=426)
         season = int(match.group('season'))
 
         season_info = next((s for s in show_info['seasons'] if s.get('season') == season), None)


Test patch: