Repository: yt-dlp/yt-dlp

Issue:
[ok.ru] Can't download non hls formats
### DO NOT REMOVE OR SKIP THE ISSUE TEMPLATE

- [X] I understand that I will be **blocked** if I *intentionally* remove or skip any mandatory\* field

### Checklist

- [X] I'm reporting that yt-dlp is broken on a **supported** site
- [X] I've verified that I'm running yt-dlp version **2023.03.04** ([update instructions](https://github.com/yt-dlp/yt-dlp#update)) or later (specify commit)
- [X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
- [X] I've checked that all URLs and arguments with special characters are [properly quoted or escaped](https://github.com/yt-dlp/yt-dlp/wiki/FAQ#video-url-contains-an-ampersand--and-im-getting-some-strange-output-1-2839-or-v-is-not-recognized-as-an-internal-or-external-command)
- [X] I've searched [known issues](https://github.com/yt-dlp/yt-dlp/issues/3766) and the [bugtracker](https://github.com/yt-dlp/yt-dlp/issues?q=) for similar issues **including closed ones**. DO NOT post duplicates
- [X] I've read the [guidelines for opening an issue](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#opening-an-issue)
- [ ] I've read about [sharing account credentials](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#are-you-willing-to-share-account-details-if-needed) and I'm willing to share it if required

### Region

_No response_

### Provide a description that is worded well enough to be understood

Can't download non hls videos.

### Provide verbose output that clearly demonstrates the problem

- [X] Run **your** yt-dlp command with **-vU** flag added (`yt-dlp -vU <your command line>`)
- [ ] If using API, add `'verbose': True` to `YoutubeDL` params instead
- [X] Copy the WHOLE output (starting with `[debug] Command-line config`) and insert it below

### Complete Verbose Output

```shell
[debug] Command-line config: ['https://ok.ru/video/1868262214220', '-f', 'full', '-vU', '--ignore-config']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: f41b949a
[debug] Python 3.8.10 (CPython x86_64 64bit) - Linux-5.15.0-71-generic-x86_64-with-glibc2.29 (OpenSSL 1.1.1f  31 Mar 2020, glibc 2.31)
[debug] exe versions: ffmpeg 4.2.7, ffprobe 4.2.7
[debug] Optional libraries: certifi-2019.11.28, secretstorage-2.3.1, sqlite3-2.6.0
[debug] Proxy map: {}
[debug] Loaded 1842 extractors
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Available version: stable@2023.03.04, Current version: stable@2023.03.04
yt-dlp is up to date (stable@2023.03.04)
[Odnoklassniki] Extracting URL: https://ok.ru/video/1868262214220
[Odnoklassniki] 1868262214220: Downloading desktop webpage
[Odnoklassniki] 1868262214220: Downloading m3u8 information
[debug] Formats sorted by: hasvid, ie_pref, lang, quality, res, fps, hdr:12(7), vcodec:vp9.2(10), channels, acodec, filesize, fs_approx, tbr, vbr, abr, asr, proto, vext, aext, hasaud, source, id
[info] 1868262214220: Downloading 1 format(s): full
[debug] Invoking http downloader on "https://vd292.mycdn.me/?expires=1685752437000&srcIp=114.10.4.70&pr=10&srcAg=CHROME&ms=185.226.52.42&type=5&sig=7FnX_Adcp0c&ct=0&urls=45.136.22.19&clientType=0&id=602557450828"
ERROR: unable to download video data: HTTP Error 400: Bad Request
Traceback (most recent call last):
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/YoutubeDL.py", line 3280, in process_info
    success, real_download = self.dl(temp_filename, info_dict)
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/YoutubeDL.py", line 3003, in dl
    return fd.download(name, new_info, subtitle)
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/downloader/common.py", line 456, in download
    ret = self.real_download(filename, info_dict)
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/downloader/http.py", line 377, in real_download
    establish_connection()
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/downloader/http.py", line 129, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "/home/nir/Code-Fu/lain-lain/yt-dlp/yt_dlp/YoutubeDL.py", line 3776, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/usr/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 400: Bad Request
```



Discussion:
When yt-dlp extracts the m3u8 formats, the CDN sets a poisonous cookie via the response headers:
```
header: Set-Cookie: tstc=p;
```
If we clear this cookie, the direct mp4 formats will be downloadable:
```diff
diff --git a/yt_dlp/extractor/odnoklassniki.py b/yt_dlp/extractor/odnoklassniki.py
index 4b73eed37..fcb117443 100644
--- a/yt_dlp/extractor/odnoklassniki.py
+++ b/yt_dlp/extractor/odnoklassniki.py
@@ -1,3 +1,5 @@
+import urllib.parse
+
 from .common import InfoExtractor
 from ..compat import (
     compat_etree_fromstring,
@@ -359,6 +361,9 @@ def _extract_desktop(self, url):
         if provider == 'LIVE_TV_APP':
             info['title'] = title
 
+        def clear_cookies(manifest_url):
+            self.cookiejar.clear(domain=urllib.parse.urlparse(manifest_url).netloc)
+
         quality = qualities(('4', '0', '1', '2', '3', '5', '6', '7'))
 
         formats = [{
@@ -372,6 +377,7 @@ def _extract_desktop(self, url):
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', 'm3u8_native',
                 m3u8_id='hls', fatal=False))
+            clear_cookies(m3u8_url)
 
         dash_manifest = metadata.get('metadataEmbedded')
         if dash_manifest:
@@ -390,6 +396,7 @@ def _extract_desktop(self, url):
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+            clear_cookies(m3u8_url)
         rtmp_url = metadata.get('rtmpUrl')
         if rtmp_url:
             formats.append({
```

Note: if anyone wants to PR this, also include this patch: https://github.com/yt-dlp/yt-dlp/issues/6442#issuecomment-1455181924

Resolving patch:
diff --git a/yt_dlp/extractor/odnoklassniki.py b/yt_dlp/extractor/odnoklassniki.py
--- a/yt_dlp/extractor/odnoklassniki.py
+++ b/yt_dlp/extractor/odnoklassniki.py
@@ -1,3 +1,5 @@
+import urllib.parse
+
 from .common import InfoExtractor
 from ..compat import (
     compat_etree_fromstring,
@@ -7,6 +9,7 @@
 )
 from ..utils import (
     ExtractorError,
+    HEADRequest,
     float_or_none,
     int_or_none,
     qualities,
@@ -15,6 +18,7 @@
     unescapeHTML,
     unified_strdate,
     unsmuggle_url,
+    url_or_none,
     urlencode_postdata,
 )
 
@@ -41,7 +45,7 @@ class OdnoklassnikiIE(InfoExtractor):
             'ext': 'mp4',
             'timestamp': 1545580896,
             'view_count': int,
-            'thumbnail': 'https://coub-attachments.akamaized.net/coub_storage/coub/simple/cw_image/c5ac87553bd/608e806a1239c210ab692/1545580913_00026.jpg',
+            'thumbnail': r're:^https?://.*\.jpg$',
             'title': 'Народная забава',
             'uploader': 'Nevata',
             'upload_date': '20181223',
@@ -65,13 +69,14 @@ class OdnoklassnikiIE(InfoExtractor):
             'title': str,
             'uploader': str,
         },
+        'skip': 'vk extractor error',
     }, {
-        # metadata in JSON
+        # metadata in JSON, webm_dash with Firefox UA
         'url': 'http://ok.ru/video/20079905452',
-        'md5': '5d2b64756e2af296e3b383a0bc02a6aa',
+        'md5': '8f477d8931c531374a3e36daec617b2c',
         'info_dict': {
             'id': '20079905452',
-            'ext': 'mp4',
+            'ext': 'webm',
             'title': 'Культура меняет нас (прекрасный ролик!))',
             'thumbnail': str,
             'duration': 100,
@@ -81,10 +86,14 @@ class OdnoklassnikiIE(InfoExtractor):
             'like_count': int,
             'age_limit': 0,
         },
+        'params': {
+            'format': 'bv[ext=webm]',
+            'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:102.0) Gecko/20100101 Firefox/102.0'},
+        },
     }, {
         # metadataUrl
         'url': 'http://ok.ru/video/63567059965189-0?fromTime=5',
-        'md5': 'f8c951122516af72e6e6ffdd3c41103b',
+        'md5': '2bae2f58eefe1b3d26f3926c4a64d2f3',
         'info_dict': {
             'id': '63567059965189-0',
             'ext': 'mp4',
@@ -98,10 +107,11 @@ class OdnoklassnikiIE(InfoExtractor):
             'age_limit': 0,
             'start_time': 5,
         },
+        'params': {'skip_download': 'm3u8'},
     }, {
         # YouTube embed (metadataUrl, provider == USER_YOUTUBE)
         'url': 'https://ok.ru/video/3952212382174',
-        'md5': '91749d0bd20763a28d083fa335bbd37a',
+        'md5': '5fb5f83ce16cb212d6bf887282b5da53',
         'info_dict': {
             'id': '5axVgHHDBvU',
             'ext': 'mp4',
@@ -116,7 +126,7 @@ class OdnoklassnikiIE(InfoExtractor):
             'live_status': 'not_live',
             'view_count': int,
             'thumbnail': 'https://i.mycdn.me/i?r=AEHujHvw2RjEbemUCNEorZbxYpb_p_9AcN2FmGik64Krkcmz37YtlY093oAM5-HIEAt7Zi9s0CiBOSDmbngC-I-k&fn=external_8',
-            'uploader_url': 'http://www.youtube.com/user/MrKewlkid94',
+            'uploader_url': 'https://www.youtube.com/@MrKewlkid94',
             'channel_follower_count': int,
             'tags': ['youtube-dl', 'youtube playlists', 'download videos', 'download audio'],
             'channel_id': 'UCVGtvURtEURYHtJFUegdSug',
@@ -145,7 +155,6 @@ class OdnoklassnikiIE(InfoExtractor):
         },
         'skip': 'Video has not been found',
     }, {
-        # TODO: HTTP Error 400: Bad Request, it only works if there's no cookies when downloading
         'note': 'Only available in mobile webpage',
         'url': 'https://m.ok.ru/video/2361249957145',
         'info_dict': {
@@ -153,8 +162,8 @@ class OdnoklassnikiIE(InfoExtractor):
             'ext': 'mp4',
             'title': 'Быковское крещение',
             'duration': 3038.181,
+            'thumbnail': r're:^https?://i\.mycdn\.me/videoPreview\?.+',
         },
-        'skip': 'HTTP Error 400',
     }, {
         'note': 'subtitles',
         'url': 'https://ok.ru/video/4249587550747',
@@ -226,6 +235,14 @@ class OdnoklassnikiIE(InfoExtractor):
         'skip': 'Site no longer embeds',
     }]
 
+    def _clear_cookies(self, cdn_url):
+        # Direct http downloads will fail if CDN cookies are set
+        # so we need to reset them after each format extraction
+        if self._get_cookies('https://notarealsubdomain.mycdn.me/'):
+            self.cookiejar.clear(domain='.mycdn.me')
+        if self._get_cookies(cdn_url):
+            self.cookiejar.clear(domain=urllib.parse.urlparse(cdn_url).hostname)
+
     @classmethod
     def _extract_embed_urls(cls, url, webpage):
         for x in super()._extract_embed_urls(url, webpage):
@@ -364,14 +381,22 @@ def _extract_desktop(self, url):
         formats = [{
             'url': f['url'],
             'ext': 'mp4',
-            'format_id': f['name'],
-        } for f in metadata['videos']]
+            'format_id': f.get('name'),
+        } for f in traverse_obj(metadata, ('videos', lambda _, v: url_or_none(v['url'])))]
 
-        m3u8_url = metadata.get('hlsManifestUrl')
+        m3u8_url = traverse_obj(metadata, 'hlsManifestUrl', 'ondemandHls')
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', 'm3u8_native',
                 m3u8_id='hls', fatal=False))
+            self._clear_cookies(m3u8_url)
+
+        for mpd_id, mpd_key in [('dash', 'ondemandDash'), ('webm', 'metadataWebmUrl')]:
+            mpd_url = metadata.get(mpd_key)
+            if mpd_url:
+                formats.extend(self._extract_mpd_formats(
+                    mpd_url, video_id, mpd_id=mpd_id, fatal=False))
+                self._clear_cookies(mpd_url)
 
         dash_manifest = metadata.get('metadataEmbedded')
         if dash_manifest:
@@ -390,6 +415,7 @@ def _extract_desktop(self, url):
         if m3u8_url:
             formats.extend(self._extract_m3u8_formats(
                 m3u8_url, video_id, 'mp4', m3u8_id='hls', fatal=False))
+            self._clear_cookies(m3u8_url)
         rtmp_url = metadata.get('rtmpUrl')
         if rtmp_url:
             formats.append({
@@ -423,6 +449,10 @@ def _extract_mobile(self, url):
             r'data-video="(.+?)"', webpage, 'json data')
         json_data = self._parse_json(unescapeHTML(json_data), video_id) or {}
 
+        redirect_url = self._request_webpage(HEADRequest(
+            json_data['videoSrc']), video_id, 'Requesting download URL').geturl()
+        self._clear_cookies(redirect_url)
+
         return {
             'id': video_id,
             'title': json_data.get('videoName'),
@@ -430,7 +460,7 @@ def _extract_mobile(self, url):
             'thumbnail': json_data.get('videoPosterSrc'),
             'formats': [{
                 'format_id': 'mobile',
-                'url': json_data.get('videoSrc'),
+                'url': redirect_url,
                 'ext': 'mp4',
             }]
         }


Test patch: