Repository: ytdl-org/youtube-dl

Issue:
Downloading from mail.ru seems broken
<!--

######################################################################
  WARNING!
  IGNORING THE FOLLOWING TEMPLATE WILL RESULT IN ISSUE CLOSED AS INCOMPLETE
######################################################################

-->


## Checklist

<!--
Carefully read and work through this check list in order to prevent the most common mistakes and misuse of youtube-dl:
- First of, make sure you are using the latest version of youtube-dl. Run `youtube-dl --version` and ensure your version is 2020.03.24. If it's not, see https://yt-dl.org/update on how to update. Issues with outdated version will be REJECTED.
- Make sure that all provided video/audio/playlist URLs (if any) are alive and playable in a browser.
- Make sure that all URLs and arguments with special characters are properly quoted or escaped as explained in http://yt-dl.org/escape.
- Search the bugtracker for similar issues: http://yt-dl.org/search-issues. DO NOT post duplicates.
- Finally, put x into all relevant boxes (like this [x])
-->

- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version **2020.03.24**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log

<!--
Provide the complete verbose output of youtube-dl that clearly demonstrates the problem.
Add the `-v` flag to your command line you run youtube-dl with (`youtube-dl -v <your command line>`), copy the WHOLE output and insert it below. It should look similar to this:
 [debug] System config: []
 [debug] User config: []
 [debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
 [debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
 [debug] youtube-dl version 2020.03.24
 [debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
 [debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
 [debug] Proxy map: {}
 <more lines>
-->

```
youtube-dl --proxy socks5://127.0.0.1:10808 --no-cache-dir -f "1080p"--merge-output-format mp4 "https://my.mail.ru/community/gotivim.mm/video/_groupvideo/288.html" -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--proxy', 'socks5://127.0.0.1:10808', '--no-cache-dir', '-f', '1080p', '--merge-output-format', 'mp4', 'https://my.mail.ru/community/gotivim.mm/video/_groupvideo/288.html', '-v']
[debug] Encodings: locale cp936, fs utf-8, out utf-8, pref cp936
[debug] youtube-dl version 2020.03.24
[debug] Python version 3.7.4 (CPython) - Windows-10-10.0.18362-SP0
[debug] exe versions: ffmpeg 4.0.2, ffprobe 4.0.2
[debug] Proxy map: {'http': 'socks5://127.0.0.1:10808', 'https': 'socks5://127.0.0.1:10808'}
[mailru] community/gotivim.mm/_groupvideo/288: Downloading webpage
[mailru] community/gotivim.mm/_groupvideo/288: Downloading video meta JSON
[debug] Invoking downloader on 'http://cdn7.my.mail.ru/hv/62465402.mp4?slave[]=s%3Ahttp%3A%2F%2Fvideo-cephgw1.i%3A8003%2Frados3%2F62465402-hv&p=f&expire_at=1585504800&touch=1499087450&reg=188&sign=db04c5f100a3c50f30e86d49a6ab5caa1b32f314'
ERROR: unable to download video data: HTTP Error 403: Forbidden
Traceback (most recent call last):
  File "c:\python37\lib\site-packages\youtube_dl\YoutubeDL.py", line 1926, in process_info
    success = dl(filename, info_dict)
  File "c:\python37\lib\site-packages\youtube_dl\YoutubeDL.py", line 1865, in dl
    return fd.download(name, info)
  File "c:\python37\lib\site-packages\youtube_dl\downloader\common.py", line 366, in download
    return self.real_download(filename, info_dict)
  File "c:\python37\lib\site-packages\youtube_dl\downloader\http.py", line 341, in real_download
    establish_connection()
  File "c:\python37\lib\site-packages\youtube_dl\downloader\http.py", line 109, in establish_connection
    ctx.data = self.ydl.urlopen(request)
  File "c:\python37\lib\site-packages\youtube_dl\YoutubeDL.py", line 2238, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "c:\python37\lib\urllib\request.py", line 531, in open
    response = meth(req, response)
  File "c:\python37\lib\urllib\request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "c:\python37\lib\urllib\request.py", line 569, in error
    return self._call_chain(*args)
  File "c:\python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "c:\python37\lib\urllib\request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden
```


## Description

<!--
Provide an explanation of your issue in an arbitrary form. Provide any additional information, suggested solution and as much context and examples as possible.
If work on your issue requires account credentials please provide them or explain how one can obtain them.
-->

It seems like mail.ru has updated their online player recently and you can't access their cdn directly to download video. Accessing CDN through browser directly returns permission denied errors as well. I've tried adding user agent and cookies but doesn't help. Is there any way to bypass this restriction? Thanks in advance.


Discussion:
You should see the #23458:
Let"s try add referer:
youtube-dl --add-header 'Referer: https://my.mail.ru/' [DASH_URL] (not work)
P.S. I studied this site. Let`s try add new cookies again(video_key,...). 
Yes, with --add-header "Cookie:video_key=da498ac65a8271fe4e5b489afff638e.........." and even without any referers download works fine.
Cookie I've got from Google Chrome for site my.mail.ru

> Yes, with --add-header "Cookie:video_key=da498ac65a8271fe4e5b489afff638e.........." and even without any referers download works fine.
> Cookie I've got from Google Chrome for site my.mail.ru

This way works fine.


Resolving patch:
diff --git a/youtube_dl/extractor/mailru.py b/youtube_dl/extractor/mailru.py
--- a/youtube_dl/extractor/mailru.py
+++ b/youtube_dl/extractor/mailru.py
@@ -128,6 +128,12 @@ def _real_extract(self, url):
                 'http://api.video.mail.ru/videos/%s.json?new=1' % video_id,
                 video_id, 'Downloading video JSON')
 
+        headers = {}
+
+        video_key = self._get_cookies('https://my.mail.ru').get('video_key')
+        if video_key:
+            headers['Cookie'] = 'video_key=%s' % video_key.value
+
         formats = []
         for f in video_data['videos']:
             video_url = f.get('url')
@@ -140,6 +146,7 @@ def _real_extract(self, url):
                 'url': video_url,
                 'format_id': format_id,
                 'height': height,
+                'http_headers': headers,
             })
         self._sort_formats(formats)
 


Test patch: