Repository: ytdl-org/youtube-dl

Issue:
[dl fail] Is netease module still being maintained?
### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2018.10.29*. If it's not, read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2018.10.29**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through the [README](https://github.com/rg3/youtube-dl/blob/master/README.md), **most notably** the [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones
- [x] Checked that provided video/audio/playlist URLs (if any) are alive and playable in a browser

### What is the purpose of your *issue*?
- [x] Bug report (encountered problems with youtube-dl)
- [ ] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

Full command and output:

```
C:\Users\inkux\Desktop>youtube-dl https://music.163.com/#/song?id=33166366 --verbose --proxy ""
[debug] System config: []
[debug] User config: ['--proxy', 'socks5://[censored]/']
[debug] Custom config: []
[debug] Command-line args: ['https://music.163.com/#/song?id=33166366', '--verbose', '--proxy', '']
[debug] Encodings: locale cp936, fs mbcs, out cp936, pref cp936
[debug] youtube-dl version 2018.10.29
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.17134
[debug] exe versions: ffmpeg N-90414-gabf35afb6f, ffprobe N-90414-gabf35afb6f
[debug] Proxy map: {}
[netease:song] 33166366: Downloading song info
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
[netease:song] 33166366: Checking song URL
[netease:song] 33166366: song URL is invalid, skipping
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpadzwnijc\build\youtube_dl\YoutubeDL.py", line 792, in extract_info
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpadzwnijc\build\youtube_dl\extractor\common.py", line 508, in extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpadzwnijc\build\youtube_dl\extractor\neteasemusic.py", line 164, in _real_extract
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmpadzwnijc\build\youtube_dl\extractor\common.py", line 1287, in _sort_formats
youtube_dl.utils.ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```
------
I've noticed from the issue page that netease module had been down for quite a while in 2016, but since I got an instruction to report this and those issues are pretty aged, I decided to report it anyway.

I was downloading this song which is totally playable in my browser(Google Chrome), and is also downloadable as a mp3 file using netease client (PC, Android), of course using a Chinese IP address.

As you can see youtube-dl correctly recognized the ID of the song from its URL but has been unable to obtain any format.

And if this module is never going to be maintained for a period of time, I think it's a good idea to disable the module if it is believed it will never work again, so no one gets a bug report request when they see netease music in your support sites list and then fails to download using yt-dl, given that netease is a pretty popular music site in China.

------


Discussion:
Well there I am. Pretty late but I randomly came across this extractor.

There are changes on netease (music.163.com). They changed endpoints and have more than one now. Also their response is different.

Song (data as in m4a) related api is
`https://music.163.com/weapi/song/enhance/player/url/v1?csrf_token=`
song details is now
`https://music.163.com/weapi/v3/song/detail?csrf_token=`
and so on. if there is enough interest I can update the extractor.

@blackjack4494 I for one would be interested. Tried to use it just now and came across this report.
Hi, I think this is still not solved, because the traceback still remains the same, how can we start fixing this? There is also a geolocation restriction in this case.
[This API](https://github.com/Binaryify/NeteaseCloudMusicApi) written in Node.js is able to get the real audio file URL with a given ID and quality. For example, https://api.moeblog.vip/163/ is a deployment of this API, and the real URL of the song with ID 29848621 can be got by `https://api.moeblog.vip/163/?type=url&id=29848621&br=128`. It worked.

Thus, examining the [Node.js API](https://github.com/Binaryify/NeteaseCloudMusicApi) will be helpful.

Geolocation restriction, like paid contents, seems to be unable to bypass, at least by this API.

I'll come back in a week after I finish my exams and try to fix this issue if no one else is going to do it.
I've translated relevant parts of the Node.js API to python, and the following script is able to get the real music file URL with a given id (needs [pycrypto](https://pypi.org/project/pycrypto/)):

```python
import requests
import json
from hashlib import md5
from Crypto.Cipher import AES
import Crypto
import time
from random import randint

HEADERS = {
    "User-Agent": "Mozilla/5.0 (Linux; U; Android 9; zh-cn; Redmi Note 8 Build/PKQ1.190616.001) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/71.0.3578.141 Mobile Safari/537.36 XiaoMi/MiuiBrowser/12.5.22",
    "Content-Type": "application/x-www-form-urlencoded",
    "Referer": "https://music.163.com",
}


KEY = "e82ckenh8dichen8"


def pad(data):
    # https://stackoverflow.com/a/10550004/12425329
    pad = 16 - len(data) % 16
    return data + pad * chr(pad)


def make_data_and_headers(song_id):
    KEY = "e82ckenh8dichen8"
    URL = "/api/song/enhance/player/url"
    cookie = {
        "osver": None,
        "deviceId": None,
        "appver": "8.0.0",
        "versioncode": "140",
        "mobilename": None,
        "buildver": "1623435496",
        "resolution": "1920x1080",
        "__csrf": "",
        "os": "pc",
        "channel": None,
        "requestId": f"{int(time.time()*1000)}_{randint(0, 1000):04}",
    }
    text = json.dumps(
        {"ids": f"[{song_id}]", "br": 999000, "header": cookie},
        separators=(",", ":"),
    )
    message = f"nobody{URL}use{text}md5forencrypt"
    m = md5()
    m.update(message.encode("latin1"))
    digest = m.hexdigest()

    data = f"{URL}-36cd479b6b5-{text}-36cd479b6b5-{digest}"

    data = '/api/song/enhance/player/url-36cd479b6b5-{"ids":"[33894312]","br":999000,"header":{"appver":"8.0.0","versioncode":"140","buildver":"1623455100","resolution":"1920x1080","__csrf":"","os":"pc","requestId":"1623455100782_0489"}}-36cd479b6b5-a036727d6cb4f68dc27d0e1962f56eb8'

    data = pad(data)

    cipher = Crypto.Cipher.AES.new(KEY, AES.MODE_ECB)
    encrypted = cipher.encrypt(data.encode("latin1"))

    headers = HEADERS
    process_v = lambda v: v if v is not None else "undefined"
    headers.update(
        {"Cookie": "; ".join([f"{k}={process_v(v)}" for [k, v] in cookie.items()])}
    )
    return (f"params={encrypted.hex().upper()}", headers)


if __name__ == "__main__":
    song_id = input("song_id? (default to 491233178)")
    if not song_id:
        song_id = 491233178
    data, headers = make_data_and_headers(491233178)
    # print(data)
    # print(headers)
    r = requests.post(
        "https://interface3.music.163.com/eapi/song/enhance/player/url",
        data=data,  # json.dumps(data, separators=(",", ":")),
        headers=HEADERS,
    )
    print(r.json())
```

The next challenge is to adapt it into youtube-dl

Resolving patch:
diff --git a/youtube_dl/aes.py b/youtube_dl/aes.py
--- a/youtube_dl/aes.py
+++ b/youtube_dl/aes.py
@@ -8,6 +8,18 @@
 BLOCK_SIZE_BYTES = 16
 
 
+def pkcs7_padding(data):
+    """
+    PKCS#7 padding
+
+    @param {int[]} data        cleartext
+    @returns {int[]}           padding data
+    """
+
+    remaining_length = BLOCK_SIZE_BYTES - len(data) % BLOCK_SIZE_BYTES
+    return data + [remaining_length] * remaining_length
+
+
 def aes_ctr_decrypt(data, key, counter):
     """
     Decrypt with aes in counter mode
@@ -76,8 +88,7 @@ def aes_cbc_encrypt(data, key, iv):
     previous_cipher_block = iv
     for i in range(block_count):
         block = data[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES]
-        remaining_length = BLOCK_SIZE_BYTES - len(block)
-        block += [remaining_length] * remaining_length
+        block = pkcs7_padding(block)
         mixed_block = xor(block, previous_cipher_block)
 
         encrypted_block = aes_encrypt(mixed_block, expanded_key)
@@ -88,6 +99,28 @@ def aes_cbc_encrypt(data, key, iv):
     return encrypted_data
 
 
+def aes_ecb_encrypt(data, key):
+    """
+    Encrypt with aes in ECB mode. Using PKCS#7 padding
+
+    @param {int[]} data        cleartext
+    @param {int[]} key         16/24/32-Byte cipher key
+    @returns {int[]}           encrypted data
+    """
+    expanded_key = key_expansion(key)
+    block_count = int(ceil(float(len(data)) / BLOCK_SIZE_BYTES))
+
+    encrypted_data = []
+    for i in range(block_count):
+        block = data[i * BLOCK_SIZE_BYTES: (i + 1) * BLOCK_SIZE_BYTES]
+        block = pkcs7_padding(block)
+
+        encrypted_block = aes_encrypt(block, expanded_key)
+        encrypted_data += encrypted_block
+
+    return encrypted_data
+
+
 def key_expansion(data):
     """
     Generate key schedule
diff --git a/youtube_dl/extractor/neteasemusic.py b/youtube_dl/extractor/neteasemusic.py
--- a/youtube_dl/extractor/neteasemusic.py
+++ b/youtube_dl/extractor/neteasemusic.py
@@ -1,20 +1,31 @@
 # coding: utf-8
 from __future__ import unicode_literals
 
-from hashlib import md5
 from base64 import b64encode
+from binascii import hexlify
 from datetime import datetime
+from hashlib import md5
+from random import randint
+import json
 import re
+import time
 
 from .common import InfoExtractor
+from ..aes import aes_ecb_encrypt, pkcs7_padding
 from ..compat import (
     compat_urllib_parse_urlencode,
     compat_str,
     compat_itertools_count,
 )
 from ..utils import (
-    sanitized_Request,
+    ExtractorError,
+    bytes_to_intlist,
     float_or_none,
+    int_or_none,
+    intlist_to_bytes,
+    sanitized_Request,
+    std_headers,
+    try_get,
 )
 
 
@@ -35,32 +46,85 @@ def _encrypt(cls, dfsid):
         result = b64encode(m.digest()).decode('ascii')
         return result.replace('/', '_').replace('+', '-')
 
+    @classmethod
+    def make_player_api_request_data_and_headers(cls, song_id, bitrate):
+        KEY = b'e82ckenh8dichen8'
+        URL = '/api/song/enhance/player/url'
+        now = int(time.time() * 1000)
+        rand = randint(0, 1000)
+        cookie = {
+            'osver': None,
+            'deviceId': None,
+            'appver': '8.0.0',
+            'versioncode': '140',
+            'mobilename': None,
+            'buildver': '1623435496',
+            'resolution': '1920x1080',
+            '__csrf': '',
+            'os': 'pc',
+            'channel': None,
+            'requestId': '{0}_{1:04}'.format(now, rand),
+        }
+        request_text = json.dumps(
+            {'ids': '[{0}]'.format(song_id), 'br': bitrate, 'header': cookie},
+            separators=(',', ':'))
+        message = 'nobody{0}use{1}md5forencrypt'.format(
+            URL, request_text).encode('latin1')
+        msg_digest = md5(message).hexdigest()
+
+        data = '{0}-36cd479b6b5-{1}-36cd479b6b5-{2}'.format(
+            URL, request_text, msg_digest)
+        data = pkcs7_padding(bytes_to_intlist(data))
+        encrypted = intlist_to_bytes(aes_ecb_encrypt(data, bytes_to_intlist(KEY)))
+        encrypted_params = hexlify(encrypted).decode('ascii').upper()
+
+        cookie = '; '.join(
+            ['{0}={1}'.format(k, v if v is not None else 'undefined')
+             for [k, v] in cookie.items()])
+
+        headers = {
+            'User-Agent': std_headers['User-Agent'],
+            'Content-Type': 'application/x-www-form-urlencoded',
+            'Referer': 'https://music.163.com',
+            'Cookie': cookie,
+        }
+        return ('params={0}'.format(encrypted_params), headers)
+
+    def _call_player_api(self, song_id, bitrate):
+        url = 'https://interface3.music.163.com/eapi/song/enhance/player/url'
+        data, headers = self.make_player_api_request_data_and_headers(song_id, bitrate)
+        try:
+            return self._download_json(
+                url, song_id, data=data.encode('ascii'), headers=headers)
+        except ExtractorError as e:
+            if type(e.cause) in (ValueError, TypeError):
+                # JSON load failure
+                raise
+        except Exception:
+            pass
+        return {}
+
     def extract_formats(self, info):
         formats = []
+        song_id = info['id']
         for song_format in self._FORMATS:
             details = info.get(song_format)
             if not details:
                 continue
-            song_file_path = '/%s/%s.%s' % (
-                self._encrypt(details['dfsId']), details['dfsId'], details['extension'])
-
-            # 203.130.59.9, 124.40.233.182, 115.231.74.139, etc is a reverse proxy-like feature
-            # from NetEase's CDN provider that can be used if m5.music.126.net does not
-            # work, especially for users outside of Mainland China
-            # via: https://github.com/JixunMoe/unblock-163/issues/3#issuecomment-163115880
-            for host in ('http://m5.music.126.net', 'http://115.231.74.139/m1.music.126.net',
-                         'http://124.40.233.182/m1.music.126.net', 'http://203.130.59.9/m1.music.126.net'):
-                song_url = host + song_file_path
+
+            bitrate = int_or_none(details.get('bitrate')) or 999000
+            data = self._call_player_api(song_id, bitrate)
+            for song in try_get(data, lambda x: x['data'], list) or []:
+                song_url = try_get(song, lambda x: x['url'])
                 if self._is_valid_url(song_url, info['id'], 'song'):
                     formats.append({
                         'url': song_url,
                         'ext': details.get('extension'),
-                        'abr': float_or_none(details.get('bitrate'), scale=1000),
+                        'abr': float_or_none(song.get('br'), scale=1000),
                         'format_id': song_format,
-                        'filesize': details.get('size'),
-                        'asr': details.get('sr')
+                        'filesize': int_or_none(song.get('size')),
+                        'asr': int_or_none(details.get('sr')),
                     })
-                    break
         return formats
 
     @classmethod
@@ -79,30 +143,16 @@ class NetEaseMusicIE(NetEaseMusicBaseIE):
     _VALID_URL = r'https?://music\.163\.com/(#/)?song\?id=(?P<id>[0-9]+)'
     _TESTS = [{
         'url': 'http://music.163.com/#/song?id=32102397',
-        'md5': 'f2e97280e6345c74ba9d5677dd5dcb45',
+        'md5': '3e909614ce09b1ccef4a3eb205441190',
         'info_dict': {
             'id': '32102397',
             'ext': 'mp3',
-            'title': 'Bad Blood (feat. Kendrick Lamar)',
+            'title': 'Bad Blood',
             'creator': 'Taylor Swift / Kendrick Lamar',
-            'upload_date': '20150517',
-            'timestamp': 1431878400,
-            'description': 'md5:a10a54589c2860300d02e1de821eb2ef',
+            'upload_date': '20150516',
+            'timestamp': 1431792000,
+            'description': 'md5:25fc5f27e47aad975aa6d36382c7833c',
         },
-        'skip': 'Blocked outside Mainland China',
-    }, {
-        'note': 'No lyrics translation.',
-        'url': 'http://music.163.com/#/song?id=29822014',
-        'info_dict': {
-            'id': '29822014',
-            'ext': 'mp3',
-            'title': '听见下雨的声音',
-            'creator': '周杰伦',
-            'upload_date': '20141225',
-            'timestamp': 1419523200,
-            'description': 'md5:a4d8d89f44656af206b7b2555c0bce6c',
-        },
-        'skip': 'Blocked outside Mainland China',
     }, {
         'note': 'No lyrics.',
         'url': 'http://music.163.com/song?id=17241424',
@@ -112,9 +162,9 @@ class NetEaseMusicIE(NetEaseMusicBaseIE):
             'title': 'Opus 28',
             'creator': 'Dustin O\'Halloran',
             'upload_date': '20080211',
+            'description': 'md5:f12945b0f6e0365e3b73c5032e1b0ff4',
             'timestamp': 1202745600,
         },
-        'skip': 'Blocked outside Mainland China',
     }, {
         'note': 'Has translated name.',
         'url': 'http://music.163.com/#/song?id=22735043',
@@ -128,7 +178,6 @@ class NetEaseMusicIE(NetEaseMusicBaseIE):
             'timestamp': 1264608000,
             'alt_title': '说出愿望吧(Genie)',
         },
-        'skip': 'Blocked outside Mainland China',
     }]
 
     def _process_lyrics(self, lyrics_info):


Test patch:
diff --git a/test/test_aes.py b/test/test_aes.py
--- a/test/test_aes.py
+++ b/test/test_aes.py
@@ -8,7 +8,7 @@
 import unittest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
-from youtube_dl.aes import aes_decrypt, aes_encrypt, aes_cbc_decrypt, aes_cbc_encrypt, aes_decrypt_text
+from youtube_dl.aes import aes_decrypt, aes_encrypt, aes_cbc_decrypt, aes_cbc_encrypt, aes_decrypt_text, aes_ecb_encrypt
 from youtube_dl.utils import bytes_to_intlist, intlist_to_bytes
 import base64
 
@@ -58,6 +58,13 @@ def test_decrypt_text(self):
         decrypted = (aes_decrypt_text(encrypted, password, 32))
         self.assertEqual(decrypted, self.secret_msg)
 
+    def test_ecb_encrypt(self):
+        data = bytes_to_intlist(self.secret_msg)
+        encrypted = intlist_to_bytes(aes_ecb_encrypt(data, self.key))
+        self.assertEqual(
+            encrypted,
+            b'\xaa\x86]\x81\x97>\x02\x92\x9d\x1bR[[L/u\xd3&\xd1(h\xde{\x81\x94\xba\x02\xae\xbd\xa6\xd0:')
+
 
 if __name__ == '__main__':
     unittest.main()