Repository: ytdl-org/youtube-dl

Issue:
soundcloud: cannot download secret embedded playlist
$ youtube-dl --verbose "https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/49676035%3Fsecret_token%3Ds-phZhg"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/49676035%3Fsecret_token%3Ds-phZhg']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.06
[debug] Python version 2.6.6 - Linux-2.6.32-431.23.3.el6.x86_64-x86_64-with-redhat-6.5-Santiago
[debug] Proxy map: {}
[soundcloud:playlist] 49676035: Downloading playlist
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
  File "/home/me/bin/youtube-dl/youtube_dl/extractor/common.py", line 211, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/home/me/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1244, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)



Discussion:
Works for me?

Works for me too. Please update youtube-dl to the last version and try again.

That's because that playlist was changed on soundcloud from private to public since the bug report. But here's another one:
% youtube-dl --verbose "https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/167464311%3Fsecret_token%3Ds-IcRLW"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/167464311%3Fsecret_token%3Ds-IcRLW']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.16.1
[debug] Python version 2.6.6 - Linux-2.6.32-431.29.2.el6.x86_64-x86_64-with-redhat-6.5-Santiago
[debug] Proxy map: {}
[soundcloud] 167464311: Downloading info JSON
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
  File "/afs/ifh.de/user/w/waschk/public/bin/youtube-dl/youtube_dl/extractor/common.py", line 211, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "/afs/ifh.de/user/w/waschk/public/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 1264, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

It is fixed now for a single track, but not for a secret playlist with more than one track, e.g. this one with a fresh checkout from git after the merge of that commit:

./youtube-dl --verbose "https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/51064053%3Fsecret_token%3Ds-OtAhG"
[debug] System config: []
[debug] User config: []
[debug] Command-line args: ['--verbose', 'https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/playlists/51064053%3Fsecret_token%3Ds-OtAhG']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2014.09.16.1
[debug] Python version 2.6.6 - Linux-2.6.32-431.29.2.el6.x86_64-x86_64-with-redhat-6.5-Santiago
[debug] Proxy map: {}
[soundcloud:playlist] 51064053: Downloading playlist
ERROR: Unable to download JSON metadata: HTTP Error 404: Not Found; please report this issue on https://yt-dl.org/bug . Be sure to call youtube-dl with the --verbose flag and include its complete output. Make sure you are using the latest version; type  youtube-dl -U  to update.
  File "./youtube-dl/youtube_dl/extractor/common.py", line 211, in _request_webpage
    return self._downloader.urlopen(url_or_request)
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 1264, in urlopen
    return self._opener.open(req, timeout=self._socket_timeout)
  File "/usr/lib64/python2.6/urllib2.py", line 397, in open
    response = meth(req, response)
  File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python2.6/urllib2.py", line 435, in error
    return self._call_chain(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain
    result = func(_args)
  File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)

On it.

Not sure what to do to create a test case for a private playlist...


Resolving patch:
diff --git a/youtube_dl/extractor/soundcloud.py b/youtube_dl/extractor/soundcloud.py
--- a/youtube_dl/extractor/soundcloud.py
+++ b/youtube_dl/extractor/soundcloud.py
@@ -238,7 +238,7 @@ def _real_extract(self, url):
 
 
 class SoundcloudSetIE(SoundcloudIE):
-    _VALID_URL = r'https?://(?:www\.)?soundcloud\.com/([\w\d-]+)/sets/([\w\d-]+)'
+    _VALID_URL = r'https?://(?:www\.)?soundcloud\.com/(?P<uploader>[\w\d-]+)/sets/(?P<slug_title>[\w\d-]+)(?:/(?P<token>[^?/]+))?'
     IE_NAME = 'soundcloud:set'
     _TESTS = [{
         'url': 'https://soundcloud.com/the-concept-band/sets/the-royal-concept-ep',
@@ -252,14 +252,19 @@ def _real_extract(self, url):
         mobj = re.match(self._VALID_URL, url)
 
         # extract uploader (which is in the url)
-        uploader = mobj.group(1)
+        uploader = mobj.group('uploader')
         # extract simple title (uploader + slug of song title)
-        slug_title = mobj.group(2)
+        slug_title = mobj.group('slug_title')
         full_title = '%s/sets/%s' % (uploader, slug_title)
+        url = 'http://soundcloud.com/%s/sets/%s' % (uploader, slug_title)
+
+        token = mobj.group('token')
+        if token:
+            full_title += '/' + token
+            url += '/' + token
 
         self.report_resolve(full_title)
 
-        url = 'http://soundcloud.com/%s/sets/%s' % (uploader, slug_title)
         resolv_url = self._resolv_url(url)
         info = self._download_json(resolv_url, full_title)
 
@@ -270,7 +275,7 @@ def _real_extract(self, url):
 
         return {
             '_type': 'playlist',
-            'entries': [self._extract_info_dict(track) for track in info['tracks']],
+            'entries': [self._extract_info_dict(track, secret_token=token) for track in info['tracks']],
             'id': info['id'],
             'title': info['title'],
         }
@@ -333,7 +338,7 @@ def _real_extract(self, url):
 
 
 class SoundcloudPlaylistIE(SoundcloudIE):
-    _VALID_URL = r'https?://api\.soundcloud\.com/playlists/(?P<id>[0-9]+)'
+    _VALID_URL = r'https?://api\.soundcloud\.com/playlists/(?P<id>[0-9]+)(?:/?\?secret_token=(?P<token>[^&]+?))$'
     IE_NAME = 'soundcloud:playlist'
     _TESTS = [
 
@@ -353,14 +358,21 @@ def _real_extract(self, url):
         playlist_id = mobj.group('id')
         base_url = '%s//api.soundcloud.com/playlists/%s.json?' % (self.http_scheme(), playlist_id)
 
-        data = compat_urllib_parse.urlencode({
+        data_dict = {
             'client_id': self._CLIENT_ID,
-        })
+        }
+        token = mobj.group('token')
+
+        if token:
+            data_dict['secret_token'] = token
+
+        data = compat_urllib_parse.urlencode(data_dict)
         data = self._download_json(
             base_url + data, playlist_id, 'Downloading playlist')
 
         entries = [
-            self._extract_info_dict(t, quiet=True) for t in data['tracks']]
+            self._extract_info_dict(t, quiet=True, secret_token=token)
+                for t in data['tracks']]
 
         return {
             '_type': 'playlist',


Test patch: