Repository: ytdl-org/youtube-dl

Issue:
Support for Minds.com
I would like to request [Minds.com](https://www.minds.com) support if you get the chance.


Discussion:
Carefully read [new issue template](https://github.com/rg3/youtube-dl/issues/new) and provide all requested information.
## Please follow the guide below

- You will be asked some questions and requested to provide some information, please read them **carefully** and answer honestly
- Put an `x` into all the boxes [ ] relevant to your *issue* (like that [x])
- Use *Preview* tab to see how your issue will actually look like

---

### Make sure you are using the *latest* version: run `youtube-dl --version` and ensure your version is *2017.05.26*. If it's not read [this FAQ entry](https://github.com/rg3/youtube-dl/blob/master/README.md#how-do-i-update-youtube-dl) and update. Issues with outdated version will be rejected.
- [x] I've **verified** and **I assure** that I'm running youtube-dl **2017.05.26**

### Before submitting an *issue* make sure you have:
- [x] At least skimmed through [README](https://github.com/rg3/youtube-dl/blob/master/README.md) and **most notably** [FAQ](https://github.com/rg3/youtube-dl#faq) and [BUGS](https://github.com/rg3/youtube-dl#bugs) sections
- [x] [Searched](https://github.com/rg3/youtube-dl/search?type=Issues) the bugtracker for similar issues including closed ones

### What is the purpose of your *issue*?
- [ ] Bug report (encountered problems with youtube-dl)
- [x] Site support request (request for adding support for a new site)
- [ ] Feature request (request for a new functionality)
- [ ] Question
- [ ] Other

---

### The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your *issue*

---

### If the purpose of this *issue* is a *bug report*, *site support request* or you are not completely sure provide the full verbose output as follows:

Add `-v` flag to **your command line** you run youtube-dl with, copy the **whole** output and insert it here. It should look similar to one below (replace it with **your** log inserted between triple ```):
```
$ youtube-dl -v <your command line>
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'http://www.youtube.com/watch?v=BaW_jenozKcj']
[debug] Encodings: locale cp1251, fs mbcs, out cp866, pref cp1251
[debug] youtube-dl version 2017.05.26
[debug] Python version 2.7.11 - Windows-2003Server-5.2.3790-SP2
[debug] exe versions: ffmpeg N-75573-g1d0487f, ffprobe N-75573-g1d0487f, rtmpdump 2.4
[debug] Proxy map: {}
...
<end of log>
```
youtube-dl -v "https://www.minds.com/archive/view/715172106794442752"
[debug] System config: []
[debug] User config: [u'--no-check-certificate', u'--output', u'/home/main/dlr/%(uploader)s/%(upload_date)s/%(title)s/%(title)s-%(id)s.%(ext)s', u'--user-agent', u'Mozilla/5.0 (compatible)', u'--audio-quality', u'0', u'--merge-output-format', u'mp4', u'--sub-lang', u'en', u'--all-subs', u'--sub-format', u'srt', u'--embed-subs', u'--write-description', u'--retries', u'10', u'--sleep-interval', u'1', u'--max-sleep-interval', u'5', u'-i', u'-f', u'(mp4)[height=360]/(mp4)[height=480]/[height=1080]/best', u'--playlist-end', u'35', u'--dateafter', u'20151104', u'--download-archive', u'/home/main/.config/youtube-dl/archive', u'--no-call-home', u'--external-downloader', u'aria2c', u'--external-downloader-args', u"--continue=true --check-integrity=true --max-connection-per-server=5 --min-split-size=1M --max-tries=0 --user-agent='Mozilla/5.0 (compatible)'"]
[debug] Custom config: []
[debug] Command-line args: [u'-v', u'https://www.minds.com/archive/view/715172106794442752']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2017.05.26
[debug] Python version 2.7.12 - Linux-4.4.0-75-generic-x86_64-with-Ubuntu-16.04-xenial
[debug] exe versions: ffmpeg 2.8.11-0ubuntu0.16.04.1, ffprobe 2.8.11-0ubuntu0.16.04.1, rtmpdump 2.4
[debug] Proxy map: {}
[generic] 715172106794442752: Requesting header
WARNING: Falling back on generic information extractor.
[generic] 715172106794442752: Downloading webpage
[generic] 715172106794442752: Extracting information
ERROR: Unsupported URL: https://www.minds.com/archive/view/715172106794442752
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 1970, in _real_extract
    doc = compat_etree_fromstring(webpage.encode('utf-8'))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2526, in compat_etree_fromstring
    doc = _XML(text, parser=etree.XMLParser(target=_TreeBuilder(element_factory=_element_factory)))
  File "/usr/local/bin/youtube-dl/youtube_dl/compat.py", line 2515, in _XML
    parser.feed(text)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1653, in feed
    self._raiseerror(v)
  File "/usr/lib/python2.7/xml/etree/ElementTree.py", line 1517, in _raiseerror
    raise err
ParseError: not well-formed (invalid token): line 32, column 41
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 760, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 433, in extract
    ie_result = self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/generic.py", line 2795, in _real_extract
    raise UnsupportedError(url)
UnsupportedError: Unsupported URL: https://www.minds.com/archive/view/715172106794442752

---

### If the purpose of this *issue* is a *site support request* please provide all kinds of example URLs support for which should be included (replace following example URLs by **yours**):
- Single video: https://www.minds.com/archive/view/715172106794442752
https://www.minds.com/newsfeed/798025111988506624
- Single video: ???
- Playlist: ???

Note that **youtube-dl does not support sites dedicated to [copyright infringement](https://github.com/rg3/youtube-dl#can-you-add-support-for-this-anime-video-site-or-site-which-shows-current-movies-for-free)**. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

---

### Description of your *issue*, suggested solution and other information

I wasn't sure I needed anything else for a site support request. Nonetheless I have provided the requested information.

I am working on this in my fork.

Resolving patch:
diff --git a/youtube_dl/extractor/extractors.py b/youtube_dl/extractor/extractors.py
--- a/youtube_dl/extractor/extractors.py
+++ b/youtube_dl/extractor/extractors.py
@@ -623,6 +623,11 @@
     MicrosoftVirtualAcademyIE,
     MicrosoftVirtualAcademyCourseIE,
 )
+from .minds import (
+    MindsIE,
+    MindsActivityIE,
+    MindsChannelIE,
+)
 from .ministrygrid import MinistryGridIE
 from .minoto import MinotoIE
 from .miomio import MioMioIE
diff --git a/youtube_dl/extractor/minds.py b/youtube_dl/extractor/minds.py
new file mode 100644
--- /dev/null
+++ b/youtube_dl/extractor/minds.py
@@ -0,0 +1,164 @@
+# coding: utf-8
+from __future__ import unicode_literals
+import re
+
+from .common import InfoExtractor
+from ..compat import compat_str
+from ..utils import (int_or_none, sanitized_Request, str_or_none,
+                     unified_strdate)
+
+
+class MindsIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?minds\.com/media/(?P<id>[0-9]+)'
+    _TEST = {
+        'url': 'https://www.minds.com/media/100000000000086822',
+        'md5': '215a658184a419764852239d4970b045',
+        'info_dict': {
+            'id': '100000000000086822',
+            'ext': 'mp4',
+            'title': 'Minds intro sequence',
+            'thumbnail': 'https://cdn-cinemr.minds.com/cinemr_com/334128440657580032/thumbnail-00001.png',
+            'uploader_id': '100000000000000341',
+            'description': '<?xml encoding="utf-8" ?>',
+            'upload_date': '20130524',
+            'timestamp': 1369404826,
+        },
+        'params': {
+            'skip_download': True,
+        },
+    }
+
+    def _real_extract(self, url):
+        video_id = self._match_id(url)
+        video_api_url = 'https://www.minds.com/api/v1/media/%s' % video_id
+        token = self._get_cookies(url).get('XSRF-TOKEN')
+        headers = {
+            'authority': 'www.minds.com',
+            'referer': url,
+            'x-xsrf-token': token.value if token else '',
+        }
+        data = self._download_json(video_api_url, video_id, headers=headers,
+                                   query={'children': 'false'})
+        formats = []
+        owner = data.get('ownerObj', {})
+
+        transcodes = data.get('transcodes', {})
+        # These keys are the width so keep the highest width last
+        keys = sorted(transcodes.keys())
+
+        for format_id in keys:
+            is_numeric = re.match('^[0-9]+\.mp4', format_id)
+            video_url = transcodes[format_id]
+            info = {
+                'url': video_url,
+                'format_id': format_id,
+                'http_headers': headers,
+            }
+            if is_numeric:
+                info['width'] = int(format_id.split('.')[0])
+            formats.append(info)
+
+        uploader_id = str_or_none(owner.get('guid') or
+                                  data.get('owner_guid') or
+                                  owner.get('legacy_guid') or
+                                  owner.get('owner_guid'))
+        description = str_or_none(data.get('description'))
+        if description:
+            description = description.strip()
+        uploader_url = age_limit = thumbnail = None
+
+        if owner.get('username'):
+            uploader_url = 'https://www.minds.com/%s' % owner.get('username')
+        if data.get('mature') is True:
+            age_limit = 18
+
+        thumbnail_api_url = data.get('thumbnail_src')
+        if thumbnail_api_url:
+            req = sanitized_Request(thumbnail_api_url)
+            req.get_method = lambda: 'HEAD'
+            res = self._request_webpage(req, video_id)
+            if res.headers.get('content-type', '').startswith('image/'):
+                thumbnail = getattr(res, 'url', None)
+        tags = data.get('tags', '').strip()
+        if isinstance(tags, compat_str) and tags:
+            tags = [x.strip() for x in tags.split(',')]
+        else:
+            tags = None
+        category = data.get('category')
+        if isinstance(category, compat_str) and category:
+            category = [category]
+        else:
+            category = None
+
+        return {
+            'id': video_id,
+            'title': data['title'],
+            'formats': formats,
+            'description': description,
+            'license': str_or_none(data.get('license')),
+            'creator': str_or_none(owner.get('name') or owner.get('username')),
+            'release_date': unified_strdate(data.get('time_created')),
+            'timestamp': int_or_none(data.get('time_created')),
+            'uploader_id': uploader_id,
+            'uploader_url': uploader_url,
+            'view_count': int_or_none(data.get('play:count')),
+            'like_count': int_or_none(data.get('thumbs:up:count')),
+            'dislike_count': int_or_none(data.get('thumbs:down:count')),
+            'average_rating': int_or_none(data.get('rating')),
+            'age_limit': age_limit,
+            'categories': [str_or_none(data.get('category'))],
+            'tags': tags,
+            # As of 20181020 the API is returning `false` for this value both
+            # at top level and within the entity.comments:count path. The only
+            # other way to get this is to fetch all comments and count.
+            'comment_count': int_or_none(data.get('comments:count')),
+            'thumbnail': thumbnail,
+        }
+
+
+class MindsActivityIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?minds\.com/newsfeed/(?P<id>[0-9]+)'
+
+    def _real_extract(self, url):
+        guid = self._match_id(url)
+        api_url = 'https://www.minds.com/api/v1/newsfeed/single/%s' % guid
+        token = self._get_cookies(url).get('XSRF-TOKEN')
+        headers = {
+            'authority': 'www.minds.com',
+            'referer': url,
+            'x-xsrf-token': token.value if token else '',
+        }
+        data = self._download_json(api_url, guid, headers=headers)
+        return self.url_result('https://www.minds.com/media/%s' % data['activity']['entity_guid'])
+
+
+class MindsChannelIE(InfoExtractor):
+    _VALID_URL = r'https?://(?:www\.)?minds\.com/(?!newsfeed|media|api)(?P<id>[^/]+)'
+
+    def _real_extract(self, url):
+        channel_name = self._match_id(url)
+        api_url = 'https://www.minds.com/api/v1/channel/%s' % channel_name
+        token = self._get_cookies(url).get('XSRF-TOKEN')
+        headers = {
+            'authority': 'www.minds.com',
+            'referer': url,
+            'x-xsrf-token': token.value if token else '',
+        }
+        data = self._download_json(api_url, channel_name, headers=headers)
+        channel = data.get('channel', {})
+        params = {'limit': 12, 'offset': ''}
+        api_url = 'https://www.minds.com/api/v1/newsfeed/personal/%s' % channel['guid']
+        entries = []
+        while True:
+            data = self._download_json(api_url, channel['guid'],
+                                       headers=headers, query=params)
+            activity = data.get('activity', [])
+            if len(activity) == 0 or not data.get('load-next'):
+                break
+            for info in activity:
+                if info.get('custom_type') != 'video':
+                    continue
+                entries.append(self.url_result('https://www.minds.com/media/%s' % info['entity_guid']))
+            params['offset'] = data['load-next']
+        return self.playlist_result(entries,
+                                    playlist_title='%s activity' % channel_name)


Test patch: