Repository: yt-dlp/yt-dlp

Issue:
CBC Gem Upgrade required
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I'm running yt-dlp version **2023.06.22** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [X] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

Canada

### Provide a description that is worded well enough to be understood

I tried to download from gem with the latest and get this error:

`yt-dlp --username 'xxx' --password 'xxx' --embed-subs --embed-metadata --embed-thumbnail -i -vU 'https://gem.cbc.ca/media/still-standing/s02e01'
[debug] Command-line config: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--embed-subs', '--embed-metadata', '--embed-thumbnail', '-i', '-vU', 'https://gem.cbc.ca/media/still-standing/s02e01']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.06.22 [https://github.com/yt-dlp/yt-dlp/commit/812cdfa06c33a40e73a8e04b3e6f42c084666a43] (darwin_exe)
[debug] Python 3.11.4 (CPython x86_64 64bit) - macOS-13.4.1-x86_64-i386-64bit (OpenSSL 1.1.1u 30 May 2023)
[debug] exe versions: ffmpeg 5.0-tessus (setts)
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1851 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.06.22, Current version: stable@2023.06.22
Current Build Hash: 6f96fe4e58ec9b70770509d4606ff4381672bffc0777df097ef96c5d2e4d4e37
yt-dlp is up to date (stable@2023.06.22)
[debug] Using fake IP 99.245.192.63 (CA) as X-Forwarded-For
[debug] Loading cbcgem.claims_token from cache
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/still-standing/s02e01
[gem.cbc.ca] still-standing/s02e01: Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
ERROR: [gem.cbc.ca] still-standing/s02e01: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using yt-dlp -U
File "yt_dlp/extractor/common.py", line 708, in extract
File "yt_dlp/extractor/cbc.py", line 360, in _real_extract
File "yt_dlp/extractor/cbc.py", line 311, in _get_claims_token
File "yt_dlp/extractor/cbc.py", line 282, in _new_claims_token
File "yt_dlp/extractor/common.py", line 1062, in download_content
File "yt_dlp/extractor/common.py", line 1026, in download_handle
File "yt_dlp/extractor/common.py", line 896, in _download_webpage_handle
File "yt_dlp/extractor/common.py", line 853, in _request_webpage

File "yt_dlp/extractor/common.py", line 835, in _request_webpage
File "yt_dlp/YoutubeDL.py", line 3786, in urlopen
File "urllib/request.py", line 525, in open
File "urllib/request.py", line 634, in http_response
File "urllib/request.py", line 563, in error
File "urllib/request.py", line 496, in _call_chain
File "urllib/request.py", line 643, in http_error_default
urllib.error.HTTPError: HTTP Error 426: Upgrade Required`

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['--username', 'PRIVATE', '--password', 'PRIVATE', '--embed-subs', '--embed-metadata', '--embed-thumbnail', '-i', '-vU', 'https://gem.cbc.ca/media/still-standing/s02e01']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.06.22 [812cdfa06] (darwin_exe)
[debug] Python 3.11.4 (CPython x86_64 64bit) - macOS-13.4.1-x86_64-i386-64bit (OpenSSL 1.1.1u  30 May 2023)
[debug] exe versions: ffmpeg 5.0-tessus (setts)
[debug] Optional libraries: Cryptodome-3.18.0, brotli-1.0.9, certifi-2023.05.07, mutagen-1.46.0, sqlite3-2.6.0, websockets-11.0.3
[debug] Proxy map: {}
[debug] Loaded 1851 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.06.22, Current version: stable@2023.06.22
Current Build Hash: 6f96fe4e58ec9b70770509d4606ff4381672bffc0777df097ef96c5d2e4d4e37
yt-dlp is up to date (stable@2023.06.22)
[debug] Using fake IP 99.227.126.127 (CA) as X-Forwarded-For
[debug] Loading cbcgem.claims_token from cache
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/still-standing/s02e01
[gem.cbc.ca] still-standing/s02e01: Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
ERROR: [gem.cbc.ca] still-standing/s02e01: Unable to download JSON metadata: HTTP Error 426: Upgrade Required (caused by <HTTPError 426: 'Upgrade Required'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "yt_dlp/extractor/common.py", line 708, in extract
  File "yt_dlp/extractor/cbc.py", line 360, in _real_extract
  File "yt_dlp/extractor/cbc.py", line 311, in _get_claims_token
  File "yt_dlp/extractor/cbc.py", line 282, in _new_claims_token
  File "yt_dlp/extractor/common.py", line 1062, in download_content
  File "yt_dlp/extractor/common.py", line 1026, in download_handle
  File "yt_dlp/extractor/common.py", line 896, in _download_webpage_handle
  File "yt_dlp/extractor/common.py", line 853, in _request_webpage

  File "yt_dlp/extractor/common.py", line 835, in _request_webpage
  File "yt_dlp/YoutubeDL.py", line 3786, in urlopen
  File "urllib/request.py", line 525, in open
  File "urllib/request.py", line 634, in http_response
  File "urllib/request.py", line 563, in error
  File "urllib/request.py", line 496, in _call_chain
  File "urllib/request.py", line 643, in http_error_default
urllib.error.HTTPError: HTTP Error 426: Upgrade Required
```



Discussion:
@makew0rld Here is my problem as a new issue as requested.
I can confirm this error. It caused by the login process, so downloading videos that don't require login will still work.
would adding `expected_status=426` to all of the calls to `https://services.radio-canada.ca/ott/cbc-api/v2/` endpoints solve this?

ref: https://github.com/yt-dlp/yt-dlp/commit/4afb208cf07b59291ae3b0c4efc83945ee5b8812
> I can confirm this error. It caused by the login process, so downloading videos that don't require login will still work.

So is there a fix for this error? I don't understand :(
@ToniWonKanobi not yet. I was just adding extra information to help with the fix of the issue.

@bashonly I just tested, and yes that will work actually. Same as last time, it's not going to work forever, but for now it does.
Here are my changes for the fix:

```patch
diff --git a/yt_dlp/extractor/cbc.py b/yt_dlp/extractor/cbc.py
index 41e092422..95e0ef31e 100644
--- a/yt_dlp/extractor/cbc.py
+++ b/yt_dlp/extractor/cbc.py
@@ -280,12 +280,12 @@ def _new_claims_token(self, email, password):
         data = json.dumps({'jwt': sig}).encode()
         headers = {'content-type': 'application/json', 'ott-device-type': 'web'}
         resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/token',
-                                   None, data=data, headers=headers)
+                                   None, data=data, headers=headers, expected_status=426)
         cbc_access_token = resp['accessToken']
 
         headers = {'content-type': 'application/json', 'ott-device-type': 'web', 'ott-access-token': cbc_access_token}
         resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/profile',
-                                   None, headers=headers)
+                                   None, headers=headers, expected_status=426)
         return resp['claimsToken']
 
     def _get_claims_token_expiry(self):

```
> Here are my changes for the fix:

Is this different from the merged fix that was mentioned previously? Because I tried that [old] fix and got the same HTTP 426 error :/
Yes this is for a different part of the code.
> Yes this is for a different part of the code.

Oh nice. Will try and then reply back prn.
Yeah, I'm still getting the same error after changing those two lines of code :/
@ToniWonKanobi I can't confirm this, the patch works for me. Can you share what command you are running? Make sure you are running from your changed copy of the code with `python3 -m yt_dlp` instead of `yt-dlp`.
> @ToniWonKanobi I can't confirm this, the patch works for me. Can you share what command you are running? Make sure you are running from your changed copy of the code with `python3 -m yt_dlp` instead of `yt-dlp`.

So I for sure wasn't using the `python3 -m` prefix, so that's one thing.

I got a little further today:

```
Anthony@Anthonys-MacBook-Air ~ % python3 -m yt_dlp --username '[redacted]' --password '[redacted]' https://gem.cbc.ca/media/heartland/s01e01 
[gem.cbc.ca] Extracting URL: https://gem.cbc.ca/media/heartland/s01e01 
[gem.cbc.ca] heartland/s01e01: Downloading JSON metadata
[gem.cbc.ca] Downloading JSON metadata
ERROR: [gem.cbc.ca] media/heartland/s01e01: Unable to download JSON metadata: HTTP Error 400: Bad Request (caused by <HTTPError 400: 'Bad Request'>); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
```


@ToniWonKanobi this seems like a different error, and I can't reproduce it either. You should open a new issue for this.

Resolving patch:
diff --git a/yt_dlp/extractor/cbc.py b/yt_dlp/extractor/cbc.py
--- a/yt_dlp/extractor/cbc.py
+++ b/yt_dlp/extractor/cbc.py
@@ -280,12 +280,12 @@ def _new_claims_token(self, email, password):
         data = json.dumps({'jwt': sig}).encode()
         headers = {'content-type': 'application/json', 'ott-device-type': 'web'}
         resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/token',
-                                   None, data=data, headers=headers)
+                                   None, data=data, headers=headers, expected_status=426)
         cbc_access_token = resp['accessToken']
 
         headers = {'content-type': 'application/json', 'ott-device-type': 'web', 'ott-access-token': cbc_access_token}
         resp = self._download_json('https://services.radio-canada.ca/ott/cbc-api/v2/profile',
-                                   None, headers=headers)
+                                   None, headers=headers, expected_status=426)
         return resp['claimsToken']
 
     def _get_claims_token_expiry(self):


Test patch: