Repository: Flexget/Flexget

Issue:
imdb_watchlist plugin broken
Presumably due to IMDB making changes on their end.

Flexget version: 2.11.8

Stacktrace:
```
2017-12-08 19:34 CRITICAL task          populate_movies_list BUG: Unhandled error in plugin imdb_watchlist: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/flexget/task.py", line 486, in __run_plugin
    return method(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/event.py", line 23, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/utils/cached_input.py", line 148, in wrapped_func
    response = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/flexget/plugins/input/imdb_watchlist.py", line 106, in on_task_input
    total_item_count = int(soup.find('div', class_='desc').get('data-size'))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
```


Discussion:
Hi. 

I have the same problem on 2.11.9 (had it on an older version aswell). Problems started yesterday. My config was working fine before,  I did not making any changes. However I dont get any errors just:

`2017-12-09 16:06 VERBOSE  imdb_watchlist find-movies     No movies were found in imdb list: watchlist
2017-12-09 16:06 WARNING  input_cache   find-movies     Input imdb_watchlist did not return a list, cannot cache.
2017-12-09 16:06 VERBOSE  details       find-movies     Task didn't produce any entries.`

also the same with ` flexget execute --now --tasks "find-movies" --no-cache`

Altho I have movies on the list. I also removed all movies and added 1. Still nothing.

Probly something to do with user created lists on imdb side?
looking at the plugin, and the returned HTML from imdb, it appears imdb changed how the page is structured, so the existing soup.find() in the plugin needs to be updated to reflect the new page layout.
I tried having a look at the code for this yesterday. It looks like the watchlist entries are no longer in the HTML returned when doing the get request, but is injected by some javascript on the webpage. I have a [gist](https://gist.github.com/jvlomax/e93158f4fdc973f116037e8ae461f615) of what the js looks like. I have no idea what would be the best way to make the imdb_watchlist plugin make use of this though.
So compact view is missing on IMDB site. So only way to get it working again is to get all data from a normal view.

Resolving patch:
diff --git a/flexget/plugins/input/imdb_watchlist.py b/flexget/plugins/input/imdb_watchlist.py
--- a/flexget/plugins/input/imdb_watchlist.py
+++ b/flexget/plugins/input/imdb_watchlist.py
@@ -1,16 +1,18 @@
 from __future__ import unicode_literals, division, absolute_import
+from builtins import * # noqa pylint: disable=unused-import, redefined-builtin
 
 import logging
+import re
 
-from requests.exceptions import HTTPError
-
+from flexget.utils.requests import RequestException
 from flexget import plugin
 from flexget.config_schema import one_or_more
 from flexget.event import event
-from flexget.utils.imdb import extract_id
+from flexget.utils.imdb import is_valid_imdb_title_id, extract_id
 from flexget.utils.cached_input import cached
 from flexget.entry import Entry
 from flexget.utils.soup import get_soup
+from flexget.utils import json
 
 
 log = logging.getLogger('imdb_watchlist')
@@ -18,16 +20,15 @@
 CUSTOM_LIST_RE = r'^ls\d{7,10}$'
 USER_LISTS = ['watchlist', 'ratings', 'checkins']
 TITLE_TYPE_MAP = {
-    'Feature': 'movies',
-    'Short Film': 'short films',
-    'Documentary': 'documentary',
-    'Video Game': 'games',
-    'Mini-Series': 'mini series',
-    'TV Series': 'shows',
-    'TV Episode': 'episodes',
-    'TV Movie': 'tv movies',
-    'TV Special': 'tv specials',
-    'Video': 'videos'
+    'movies': 'movie',
+    'short films': 'short',
+    'games': 'videoGame',
+    'mini series': 'tvMiniSeries',
+    'shows': 'tvSeries',
+    'episodes': 'tvEpisode',
+    'tv movies': 'tvMovie',
+    'tv specials': 'tvSpecial',
+    'videos': 'video'
 }
 
 
@@ -55,7 +56,7 @@ class ImdbWatchlist(object):
                  'default': 'en-us'},
             'type': {
                 'oneOf': [
-                    one_or_more({'type': 'string', 'enum': list(TITLE_TYPE_MAP.values())}, unique_items=True),
+                    one_or_more({'type': 'string', 'enum': list(TITLE_TYPE_MAP.keys())}, unique_items=True),
                     {'type': 'string', 'enum': ['all']}
                 ]}
         },
@@ -84,72 +85,139 @@ def on_task_input(self, task, config):
         # Create movie entries by parsing imdb list page(s) html using beautifulsoup
         log.verbose('Retrieving imdb list: %s', config['list'])
 
-        params = {'view': 'compact', 'start': 1}
+        headers = {'Accept-Language': config.get('force_language')}
+        params = {'view': 'detail', 'page': 1}
         if config['list'] in ['watchlist', 'ratings', 'checkins']:
             url = 'http://www.imdb.com/user/%s/%s' % (config['user_id'], config['list'])
+            if config['list'] == 'watchlist':
+                params = {'view': 'detail'}
         else:
             url = 'http://www.imdb.com/list/%s' % config['list']
+        if 'all' not in config['type']:
+            title_types = [TITLE_TYPE_MAP[title_type] for title_type in config['type']]
+            params['title_type'] = ','.join(title_types)
+            params['sort'] = 'list_order%2Casc'
 
-        headers = {'Accept-Language': config.get('force_language')}
-        log.debug('Requesting: %s %s', url, headers)
+        if config['list'] == 'watchlist':
+            entries = self.parse_react_widget(task, config, url, params, headers)
+        else:
+            entries = self.parse_html_list(task, config, url, params, headers)
+        return entries
 
+    def fetch_page(self, task, url, params, headers):
+        log.debug('Requesting: %s %s', url, headers)
         try:
             page = task.requests.get(url, params=params, headers=headers)
-        except HTTPError as e:
-            raise plugin.PluginError(e.args[0])
+        except RequestException as e:
+            raise plugin.PluginError(str(e))
         if page.status_code != 200:
-            raise plugin.PluginError('Unable to get imdb list. Either list is private or does not exist.')
+            raise plugin.PluginError(
+                'Unable to get imdb list. Either list is private or does not exist.' +
+                ' Html status code was: %d.' % page.status_code)
+        return page
 
-        soup = get_soup(page.text)
+    def parse_react_widget(self, task, config, url, params, headers):
+        page = self.fetch_page(task, url, params, headers)
+        try:
+            json_vars = json.loads(re.search('IMDbReactInitialState.push\((.+?)\);\n', page.text).group(1))
+        except (TypeError, AttributeError, ValueError) as e:
+            raise plugin.PluginError(
+                'Unable to get imdb list from imdb react widget.' +
+                ' Either the list is empty or the imdb parser of the imdb_watchlist plugin is broken.' +
+                ' Original error: %s.' % str(e))
+        total_item_count = 0
+        if 'list' in json_vars and 'items' in json_vars['list']:
+            total_item_count = len(json_vars['list']['items'])
+        if not total_item_count:
+            log.verbose('No movies were found in imdb list: %s', config['list'])
+            return
+        imdb_ids = []
+        for item in json_vars['list']['items']:
+            if is_valid_imdb_title_id(item.get('const')):
+                imdb_ids.append(item['const'])
+        params = {'ids': ','.join(imdb_ids)}
+        url = 'http://www.imdb.com/title/data'
+        try:
+            json_data = self.fetch_page(task, url, params, headers).json()
+        except (ValueError, TypeError) as e:
+            raise plugin.PluginError('Unable to get imdb list from imdb JSON API.' +
+                ' Either the list is empty or the imdb parser of the imdb_watchlist plugin is broken.' +
+                ' Original error: %s.' % str(e))
+        log.verbose('imdb list contains %d items', len(json_data))
+        log.debug('First entry (imdb id: %s) looks like this: %s', imdb_ids[0], json_data[imdb_ids[0]])
+        entries = []
+        for imdb_id in imdb_ids:
+            entry = Entry()
+            if not ('title' in json_data[imdb_id] and
+                    'primary' in json_data[imdb_id]['title'] and
+                    'href' in json_data[imdb_id]['title']['primary'] and
+                    'title' in json_data[imdb_id]['title']['primary']):
+                log.debug('no title or link found for item %s, skipping', imdb_id)
+                continue
+            if 'type' in json_data[imdb_id]['title']:
+                entry['imdb_type'] = json_data[imdb_id]['title']['type']
+            title = json_data[imdb_id]['title']['primary']['title']
+            entry['title'] = title
+            if 'year' in json_data[imdb_id]['title']['primary']:
+                year = json_data[imdb_id]['title']['primary']['year'][0]
+                entry['title'] += ' (%s)' % year
+                entry['imdb_year'] = year
+            entry['url'] = json_data[imdb_id]['title']['primary']['href']
+            entry['imdb_id'] = imdb_id
+            entry['imdb_name'] = entry['title']
+            entries.append(entry)
+        return entries
 
+    def parse_html_list(self, task, config, url, params, headers):
+        page = self.fetch_page(task, url, params, headers)
+        soup = get_soup(page.text)
         try:
-            total_item_count = int(soup.find('div', class_='desc').get('data-size'))
-            log.verbose('imdb list contains %s items', total_item_count)
+            item_text = soup.find('div', class_='lister-total-num-results').string.split()
+            total_item_count = int(item_text[0])
+            log.verbose('imdb list contains %d items', total_item_count)
         except AttributeError:
             total_item_count = 0
-        except ValueError as e:
+        except (ValueError, TypeError) as e:
             # TODO Something is wrong if we get a ValueError, I think
-            raise plugin.PluginError('Received invalid movie count: %s - %s' %
-                                     (soup.find('div', class_='desc').get('data-size'), e))
+            raise plugin.PluginError('Received invalid movie count: %s ; %s' %
+                                     (soup.find('div', class_='lister-total-num-results').string, e))
 
-        if total_item_count == 0:
+        if not total_item_count:
             log.verbose('No movies were found in imdb list: %s', config['list'])
             return
 
         entries = []
         items_processed = 0
+        page_no = 1
         while items_processed < total_item_count:
             # Fetch the next page unless we've just begun
             if items_processed:
-                params['start'] = items_processed + 1
-                page = task.requests.get(url, params=params)
-                if page.status_code != 200:
-                    raise plugin.PluginError('Unable to get imdb list.')
+                page_no += 1
+                params['page'] = page_no
+                page = self.fetch_page(task, url, params, headers)
                 soup = get_soup(page.text)
 
-            items = soup.find_all(attrs={'data-item-id': True, 'class': 'list_item'})
+            items = soup.find_all('div', class_='lister-item')
+            if not items:
+                log.debug('no items found on page: %s, aborting.', url)
+                break
+            log.debug('%d items found on page %d', len(items), page_no)
 
             for item in items:
                 items_processed += 1
-                a = item.find('td', class_='title').find('a')
+                a = item.find('h3', class_='lister-item-header').find('a')
                 if not a:
                     log.debug('no title link found for row, skipping')
                     continue
 
-                title_type = item.find('td', class_='title_type').text.strip()
-                if 'all' not in config['type'] and TITLE_TYPE_MAP.get(title_type) not in config['type']:
-                    log.verbose('Skipping title %s since it is a %s', a.text,
-                                TITLE_TYPE_MAP.get(title_type).rstrip('s'))
-                    continue
                 link = ('http://www.imdb.com' + a.get('href')).rstrip('/')
                 entry = Entry()
                 entry['title'] = a.text
-                entry['imdb_type'] = title_type
                 try:
-                    year = int(item.find('td', class_='year').text)
+                    year = int(item.find('span', class_='lister-item-year').text)
                     entry['title'] += ' (%s)' % year
                     entry['imdb_year'] = year
-                except ValueError:
+                except (ValueError, TypeError):
                     pass
                 entry['url'] = link
                 entry['imdb_id'] = extract_id(link)


Test patch: