Repository: ytdl-org/youtube-dl

Issue:
[cammodels] ExtractorError: Unable to find manifest URL root


- [x] I'm reporting a broken site support
- [x] I've verified that I'm running youtube-dl version **2019.10.16**
- [x] I've checked that all provided URLs are alive and playable in a browser
- [x] I've checked that all URLs and arguments with special characters are properly quoted or escaped
- [x] I've searched the bugtracker for similar issues including closed ones


## Verbose log
LATEST VERSION (broken):
```
# ./youtube-dl -vg https://www.cammodels.com/cam/agnesss
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vg', u'https://www.cammodels.com/cam/agnesss']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.10.16
[debug] Python version 2.7.16 (CPython) - Linux-5.3.6-050306-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3, rtmpdump 2.4
[debug] Proxy map: {}
ERROR: Unable to find manifest URL root; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "./youtube-dl/youtube_dl/YoutubeDL.py", line 796, in extract_info
    ie_result = ie.extract(url)
  File "./youtube-dl/youtube_dl/extractor/common.py", line 530, in extract
    ie_result = self._real_extract(url)
  File "./youtube-dl/youtube_dl/extractor/cammodels.py", line 43, in _real_extract
    raise ExtractorError(error, expected=expected)
ExtractorError: Unable to find manifest URL root; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
```
LAST KNOWN WORKING:
```
# ./youtube-dl-lastok -vg https://www.cammodels.com/cam/agnesss
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: [u'-vg', u'https://www.cammodels.com/cam/agnesss']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2019.06.27
[debug] Python version 2.7.16 (CPython) - Linux-5.3.6-050306-generic-x86_64-with-Ubuntu-19.04-disco
[debug] exe versions: ffmpeg 4.1.3, ffprobe 4.1.3, rtmpdump 2.4
[debug] Proxy map: {}
[debug] Default format spec: bestvideo+bestaudio/best
rtmp://sea1c-edge-33.naiadsystems.com:1936/live/01ead87f-66cc-465c-9cca-5f91e2505708_2000_1280x720_56
```


## Description
* You need to use a model that is online: change the name in the URL accordingly
* One could make it work again by reverting the changes in utils.py ("random_user_agent"):
```
utils.py, line 1669:
   > 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:59.0) Gecko/20100101 Firefox/59.0',
   < 'User-Agent': random_user_agent(),
 
```
* not sure if OS updates or web site changes made a difference because it has been working with a version of youtube-dl  _2019.06.27 < ok < 2019.09.28_ in September where utils.py was already the newer version
* no difference in using python2 or python3

Thanks for your work.


Discussion:

Resolving patch:
diff --git a/youtube_dl/extractor/cammodels.py b/youtube_dl/extractor/cammodels.py
--- a/youtube_dl/extractor/cammodels.py
+++ b/youtube_dl/extractor/cammodels.py
@@ -3,7 +3,6 @@
 
 from .common import InfoExtractor
 from ..utils import (
-    ExtractorError,
     int_or_none,
     url_or_none,
 )
@@ -20,32 +19,11 @@ class CamModelsIE(InfoExtractor):
     def _real_extract(self, url):
         user_id = self._match_id(url)
 
-        webpage = self._download_webpage(
-            url, user_id, headers=self.geo_verification_headers())
-
-        manifest_root = self._html_search_regex(
-            r'manifestUrlRoot=([^&\']+)', webpage, 'manifest', default=None)
-
-        if not manifest_root:
-            ERRORS = (
-                ("I'm offline, but let's stay connected", 'This user is currently offline'),
-                ('in a private show', 'This user is in a private show'),
-                ('is currently performing LIVE', 'This model is currently performing live'),
-            )
-            for pattern, message in ERRORS:
-                if pattern in webpage:
-                    error = message
-                    expected = True
-                    break
-            else:
-                error = 'Unable to find manifest URL root'
-                expected = False
-            raise ExtractorError(error, expected=expected)
-
         manifest = self._download_json(
-            '%s%s.json' % (manifest_root, user_id), user_id)
+            'https://manifest-server.naiadsystems.com/live/s:%s.json' % user_id, user_id)
 
         formats = []
+        thumbnails = []
         for format_id, format_dict in manifest['formats'].items():
             if not isinstance(format_dict, dict):
                 continue
@@ -85,6 +63,13 @@ def _real_extract(self, url):
                         'preference': -1,
                     })
                 else:
+                    if format_id == 'jpeg':
+                        thumbnails.append({
+                            'url': f['url'],
+                            'width': f['width'],
+                            'height': f['height'],
+                            'format_id': f['format_id'],
+                        })
                     continue
                 formats.append(f)
         self._sort_formats(formats)
@@ -92,6 +77,7 @@ def _real_extract(self, url):
         return {
             'id': user_id,
             'title': self._live_title(user_id),
+            'thumbnails': thumbnails,
             'is_live': True,
             'formats': formats,
             'age_limit': 18


Test patch: