Repository: pymedusa/Medusa

Issue:
TorrentLeech provider not working correctly after their redesign
TL recently did a redesign. As a result I *think* some of their category numbers/APIs have changed and this is causing issues finding shows. Medusa either can no longer find anything that I know exists in TL or can only find 1080p (even when I know a 720p version exists).

**Branch/Commit:** 77b20916ab577b82112ebc624f816054788c63f2
**OS:** Fedora 27
**What you did:** Initiated a manual search for an episode I know exists on TorrentLeech
**What happened:** It will either return no results due to the below error or it will return only 1080p results when I know that 720p exists

**Logs:**
```
2018-04-23 21:13:12 INFO     SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Performing episode search for Family Guy
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Search mode: Episode
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Search string: Family Guy S16E16
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] GET URL: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44 [Status: 503]
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] User-Agent: Medusa/0.2.2 (Linux; 4.15.17-300.fc27.x86_64; 1fa73858-4731-11e8-9b9c-9e0a979a5796)
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] CloudFlare protection detected, trying to bypass it
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] GET URL: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44 [Status: 503]
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] The response returned a non-200 response while requesting url https://classic.torrentleech.org/torrents/browse. Error: HTTPError(u'503 Server Error: Service Temporarily Unavailable for url: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44',)
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No data returned from provider
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found 0 unique items
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found qualities: []
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No unknown qualities in results
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Fallback season pack search initiate
2018-04-23 21:13:21 INFO     SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Performing season pack search for Family Guy
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found 0 unique items
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found qualities: []
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No unknown qualities in results
```



Discussion:
@rssfed23 why don't you use Jackett?
I can probably fix the provider + test it, I just need some time to work on it.
Same issue here

Resolving patch:
diff --git a/medusa/providers/torrent/__init__.py b/medusa/providers/torrent/__init__.py
--- a/medusa/providers/torrent/__init__.py
+++ b/medusa/providers/torrent/__init__.py
@@ -35,7 +35,6 @@
     torrent9,
     torrentbytes,
     torrenting,
-    torrentleech,
     tvchaosuk,
     yggtorrent,
     zooqle,
@@ -48,6 +47,7 @@
     norbits,
     rarbg,
     torrentday,
+    torrentleech,
     xthor,
 )
 from medusa.providers.torrent.rss import (
diff --git a/medusa/providers/torrent/html/torrentleech.py b/medusa/providers/torrent/html/torrentleech.py
deleted file mode 100644
--- a/medusa/providers/torrent/html/torrentleech.py
+++ /dev/null
@@ -1,202 +0,0 @@
-# coding=utf-8
-
-"""Provider code for TorrentLeech."""
-
-from __future__ import unicode_literals
-
-import logging
-
-from medusa import tv
-from medusa.bs4_parser import BS4Parser
-from medusa.helper.common import (
-    convert_size,
-)
-from medusa.logger.adapters.style import BraceAdapter
-from medusa.providers.torrent.torrent_provider import TorrentProvider
-
-from requests.compat import urljoin
-from requests.utils import dict_from_cookiejar
-
-log = BraceAdapter(logging.getLogger(__name__))
-log.logger.addHandler(logging.NullHandler())
-
-
-class TorrentLeechProvider(TorrentProvider):
-    """TorrentLeech Torrent provider."""
-
-    def __init__(self):
-        """Initialize the class."""
-        super(TorrentLeechProvider, self).__init__('TorrentLeech')
-
-        # Credentials
-        self.username = None
-        self.password = None
-
-        # URLs
-        self.url = 'https://classic.torrentleech.org'
-        self.urls = {
-            'login': urljoin(self.url, 'user/account/login'),
-            'search': urljoin(self.url, 'torrents/browse'),
-        }
-
-        # Proper Strings
-        self.proper_strings = ['PROPER', 'REPACK', 'REAL', 'RERIP']
-
-        # Miscellaneous Options
-
-        # Torrent Stats
-        self.minseed = None
-        self.minleech = None
-
-        # Cache
-        self.cache = tv.Cache(self)
-
-    def search(self, search_strings, age=0, ep_obj=None, **kwargs):
-        """
-        Search a provider and parse the results.
-
-        :param search_strings: A dict with mode (key) and the search value (value)
-        :param age: Not used
-        :param ep_obj: Not used
-        :returns: A list of search results (structure)
-        """
-        results = []
-        if not self.login():
-            return results
-
-        # TV, Episodes, BoxSets, Episodes HD, Animation, Anime, Cartoons, Foreign
-        # 2,26,27,32,7,34,35,44
-
-        for mode in search_strings:
-            log.debug('Search mode: {0}', mode)
-
-            for search_string in search_strings[mode]:
-
-                if mode != 'RSS':
-                    log.debug('Search string: {search}',
-                              {'search': search_string})
-
-                    categories = ['2', '7', '35']
-                    categories += ['26', '32', '44'] if mode == 'Episode' else ['27']
-                    if self.series and self.series.is_anime:
-                        categories += ['34']
-                else:
-                    categories = ['2', '26', '27', '32', '7', '34', '35', '44']
-
-                search_params = {
-                    'categories': ','.join(categories),
-                    'query': search_string
-                }
-                response = self.session.get(self.urls['search'], params=search_params)
-                if not response or not response.text:
-                    log.debug('No data returned from provider')
-                    continue
-
-                results += self.parse(response.text, mode)
-
-        return results
-
-    def parse(self, data, mode):
-        """
-        Parse search results for items.
-
-        :param data: The raw response from a search
-        :param mode: The current mode used to search, e.g. RSS
-
-        :return: A list of items found
-        """
-        def process_column_header(td):
-            result = ''
-            if td.a:
-                result = td.a.get('title')
-            if not result:
-                result = td.get_text(strip=True)
-            return result
-
-        items = []
-
-        with BS4Parser(data, 'html5lib') as html:
-            torrent_table = html.find('table', id='torrenttable')
-            torrent_rows = torrent_table('tr') if torrent_table else []
-
-            # Continue only if at least one release is found
-            if len(torrent_rows) < 2:
-                log.debug('Data returned from provider does not contain any torrents')
-                return items
-
-            labels = [process_column_header(label) for label in torrent_rows[0]('th')]
-
-            # Skip column headers
-            for row in torrent_rows[1:]:
-                cells = row('td')
-
-                try:
-                    name = cells[labels.index('Name')]
-                    title = name.find('a').get_text(strip=True)
-                    download_url = row.find('td', class_='quickdownload').find('a')
-                    if not all([title, download_url]):
-                        continue
-
-                    download_url = urljoin(self.url, download_url['href'])
-
-                    seeders = int(cells[labels.index('Seeders')].get_text(strip=True))
-                    leechers = int(cells[labels.index('Leechers')].get_text(strip=True))
-
-                    # Filter unseeded torrent
-                    if seeders < min(self.minseed, 1):
-                        if mode != 'RSS':
-                            log.debug("Discarding torrent because it doesn't meet the"
-                                      " minimum seeders: {0}. Seeders: {1}",
-                                      title, seeders)
-                        continue
-
-                    torrent_size = cells[labels.index('Size')].get_text()
-                    size = convert_size(torrent_size) or -1
-
-                    pubdate_raw = name.get_text(strip=True)[-19:]
-                    pubdate = self.parse_pubdate(pubdate_raw)
-
-                    item = {
-                        'title': title,
-                        'link': download_url,
-                        'size': size,
-                        'seeders': seeders,
-                        'leechers': leechers,
-                        'pubdate': pubdate,
-                    }
-                    if mode != 'RSS':
-                        log.debug('Found result: {0} with {1} seeders and {2} leechers',
-                                  title, seeders, leechers)
-
-                    items.append(item)
-                except (AttributeError, TypeError, KeyError, ValueError, IndexError):
-                    log.exception('Failed parsing provider.')
-
-        return items
-
-    def login(self):
-        """Login method used for logging in before doing search and torrent downloads."""
-        cookies = dict_from_cookiejar(self.session.cookies)
-        if any(cookies.values()) and cookies.get('member_id'):
-            return True
-
-        login_params = {
-            'username': self.username,
-            'password': self.password,
-            'login': 'submit',
-            'remember_me': 'on',
-        }
-
-        response = self.session.post(self.urls['login'], data=login_params)
-        if not response or not response.text:
-            log.warning('Unable to connect to provider')
-            return False
-
-        if '<title>Login :: TorrentLeech.org</title>' in response.text:
-            log.warning('Invalid username or password. Check your settings')
-            return False
-
-        return True
-
-
-provider = TorrentLeechProvider()
diff --git a/medusa/providers/torrent/json/torrentleech.py b/medusa/providers/torrent/json/torrentleech.py
new file mode 100644
--- /dev/null
+++ b/medusa/providers/torrent/json/torrentleech.py
@@ -0,0 +1,246 @@
+# coding=utf-8
+
+"""Provider code for TorrentLeech."""
+
+from __future__ import division, unicode_literals
+
+import logging
+import math
+
+from medusa import tv
+from medusa.helper.common import convert_size
+from medusa.logger.adapters.style import BraceAdapter
+from medusa.providers.torrent.torrent_provider import TorrentProvider
+
+from requests.compat import urljoin
+from requests.utils import dict_from_cookiejar
+
+log = BraceAdapter(logging.getLogger(__name__))
+log.logger.addHandler(logging.NullHandler())
+
+
+class TorrentLeechProvider(TorrentProvider):
+    """TorrentLeech Torrent provider."""
+
+    def __init__(self):
+        """Initialize the class."""
+        super(TorrentLeechProvider, self).__init__('TorrentLeech')
+
+        # Credentials
+        self.username = None
+        self.password = None
+
+        # URLs
+        self.url = 'https://www.torrentleech.org'
+        self.urls = {
+            'login': urljoin(self.url, 'user/account/login'),
+            'search': urljoin(self.url, 'torrents/browse/list/'),
+            'download': urljoin(self.url, 'download/{id}/{file}'),
+            'details': urljoin(self.url, 'torrent/{id}'),
+        }
+
+        # Proper Strings
+        self.proper_strings = ['PROPER', 'REPACK', 'REAL', 'RERIP']
+
+        # Miscellaneous Options
+        self.max_torrents = 100
+
+        # Torrent Stats
+        self.minseed = None
+        self.minleech = None
+
+        # Cache
+        self.cache = tv.Cache(self)
+
+    def search(self, search_strings, age=0, ep_obj=None, **kwargs):
+        """
+        Search a provider and parse the results.
+
+        :param search_strings: A dict with mode (key) and the search value (value)
+        :param age: Not used
+        :param ep_obj: Not used
+        :returns: A list of search results (structure)
+        """
+        results = []
+        if not self.login():
+            return results
+
+        # Categories:
+        #   TV
+        #     26: Episodes
+        #     27: BoxSets
+        #     32: Episodes HD
+        #   Animation
+        #     34: Anime
+        #     35: Cartoons
+        #   Foreign
+        #     44: Foreign
+
+        for mode in search_strings:
+            log.debug('Search mode: {0}', mode)
+
+            # Configured for: mode == 'RSS'
+            search_params = {
+                'categories': ['26', '27', '32', '34', '35', '44'],
+                'query': '',
+                'orderby': 'added',
+                'order': 'desc',
+            }
+
+            for search_string in search_strings[mode]:
+                if mode != 'RSS':
+                    log.debug('Search string: {search}',
+                              {'search': search_string})
+
+                    search_params['categories'] = ['35']
+                    search_params['categories'] += ['26', '32', '44'] if mode == 'Episode' else ['27']
+                    if self.series and self.series.is_anime:
+                        search_params['categories'] += ['34']
+
+                    search_params['query'] = search_string
+                    search_params['orderby'] = 'seeders'
+
+                fragments = 'categories/{categories}/'
+                if search_params['query']:
+                    fragments += 'query/{query}/'
+                fragments += 'orderby/{orderby}/order/{order}/'
+
+                path_params = fragments.format(
+                    categories=','.join(search_params['categories']),
+                    query=search_params['query'],
+                    orderby=search_params['orderby'],
+                    order=search_params['order']
+                )
+                search_url = urljoin(self.urls['search'], path_params)
+
+                data = self.session.get_json(search_url)
+                if not data:
+                    log.debug('No data returned from provider')
+                    continue
+
+                results += self.parse(data, mode)
+
+                # Handle pagination
+                results += self._pagination(data, mode, search_url)
+
+        return results
+
+    def parse(self, data, mode):
+        """
+        Parse search results for items.
+
+        :param data: The raw response from a search
+        :param mode: The current mode used to search, e.g. RSS
+
+        :return: A list of items found
+        """
+        items = []
+
+        torrents = data['torrentList']
+        user_timezone = data.get('userTimeZone', 'UTC')
+
+        # Continue only if at least one release is found
+        if not torrents:
+            log.debug('Data returned from provider does not contain any torrents')
+            return items
+
+        for torrent in torrents:
+            try:
+                title = torrent['name']
+                download_url = self.urls['download'].format(id=torrent['fid'], file=torrent['filename'])
+
+                seeders = int(torrent['seeders'])
+                leechers = int(torrent['leechers'])
+
+                # Filter unseeded torrent
+                if seeders < min(self.minseed, 1):
+                    if mode != 'RSS':
+                        log.debug("Discarding torrent because it doesn't meet the"
+                                  " minimum seeders: {0}. Seeders: {1}", title, seeders)
+                    continue
+
+                size = convert_size(torrent['size']) or -1
+
+                pubdate_raw = torrent['addedTimestamp']
+                pubdate = self.parse_pubdate(pubdate_raw, timezone=user_timezone)
+
+                item = {
+                    'title': title,
+                    'link': download_url,
+                    'size': size,
+                    'seeders': seeders,
+                    'leechers': leechers,
+                    'pubdate': pubdate,
+                }
+                if mode != 'RSS':
+                    log.debug('Found result: {0} with {1} seeders and {2} leechers',
+                              title, seeders, leechers)
+
+                items.append(item)
+            except (AttributeError, TypeError, KeyError, ValueError, IndexError):
+                log.exception('Failed parsing provider.')
+
+        return items
+
+    def login(self):
+        """Login method used for logging in before doing search and torrent downloads."""
+        cookies = dict_from_cookiejar(self.session.cookies)
+        if any(cookies.values()) and cookies.get('member_id'):
+            return True
+
+        login_params = {
+            'username': self.username,
+            'password': self.password,
+            'login': 'submit',
+            'remember_me': 'on',
+        }
+
+        response = self.session.post(self.urls['login'], data=login_params)
+        if not response or not response.text:
+            log.warning('Unable to connect to provider')
+            return False
+
+        if '<title>Login :: TorrentLeech.org</title>' in response.text:
+            log.warning('Invalid username or password. Check your settings')
+            return False
+
+        return True
+
+    def _pagination(self, data, mode, search_url):
+        """If needed, query the next page(s) of results, parse them and return."""
+        results = []
+        num_found = data.get('numFound', 0)
+        per_page = data.get('perPage', 35)
+
+        if per_page < 100 and num_found > per_page:
+            log.warning('It is recommended to change "Default Results Per Page" to 100'
+                        ' in your profile options on {name}.', {'name': self.name})
+
+        try:
+            pages = math.ceil(self.max_torrents / per_page)
+        except ZeroDivisionError:
+            pages = 1
+
+        if num_found and num_found > per_page and pages > 1:
+            log.debug('Total results found: {total}, getting {pages} more page{s} of results', {
+                'total': num_found,
+                'pages': pages - 1,
+                's': ('', 's')[pages - 1 > 1]
+            })
+
+            for page in range(2, pages + 1):
+                page_url = urljoin(search_url, 'page/{page}/'.format(page=page))
+                data = self.session.get_json(page_url)
+
+                if not data:
+                    log.debug('Page {0} returned no data from provider,'
+                              ' not getting more pages.', page)
+                    break
+
+                log.debug('Parsing page {0} of results', page)
+                results += self.parse(data, mode)
+
+        return results
+
+
+provider = TorrentLeechProvider()


Test patch: