Repository: Flexget/Flexget

Issue:
imdb_list plugin not detecting movies
### Expected behaviour:
When using imdb_list plugin, the movies from IMDB list are retrieved and added to movie_list (in my case).

### Actual behaviour:
When using imdb_list plugin, the movies from IMDB list are retrieved but not parsed (see output log).
This might be related to issue #2035 on imdb_watchlist plugin.

##### Sample log:
```
DEBUG    imdb_list     queue_from_imdb parsing line from csv: 27, tt5052474, 2017-12-24, 2017-12-24, , Sicario 2: Soldado, http://www.imdb.com/title/tt5052474/, movie, , , 2018, Action, Crime, Drama, Thriller, , 2018-06-28, Stefano Sollima
DEBUG    imdb_list     queue_from_imdb no movie row detected, skipping. 27, tt5052474, 2017-12-24, 2017-12-24, , Sicario 2: Soldado, http://www.imdb.com/title/tt5052474/, movie, , , 2018, Action, Crime, Drama, Thriller, , 2018-06-28, Stefano Sollima
```

#### Config:
```
templates:
  dl:
    transmission:
      host: ...
      port: ...
      path: ...

tasks:
  queue_from_imdb:
    imdb_list:
      login: ...
      password: ...
      list: watchlist
      force_language: en-us
    accept_all: yes
    seen: local
    list_add:
      - movie_list: movies

  movies:
    rss: ...
    quality: 720p-1080p
    list_match:
      from:
        - movie_list: movies
    template: dl

schedules:
  - tasks: '*'
    interval:
      minutes: 30
```
  
#### Log:
manual execution output log: https://pastebin.com/mPvyr9sy

### Additional information:

- FlexGet version: 2.11.18
- Python version: 3.6.0
- Installation method: pip
- Using daemon (yes/no): yes
- OS and version: QNAP TS 4.3.4



Discussion:
This plugin is not really being maintained anymore due to captchas and whatnot.
I still think we should provide a command to spin up web server with a login page to complete the login. Session will practically never expire ... requires quite a bit effort to do it though :(
I had the same issue and posted in issue #2035

Debug log 
https://pastebin.com/ateXfqd4

I've been dabbling with selenium at work recently, I'll look into using that 
Selenium will not work very well on headless systems. It will be quite complicated for the average user. Time is better spent elsewhere
If you decide to fix the login issues, it may make sense to migrate the working list parsing functions from imdb_watchlist to [utils/imdb.py](https://github.com/Flexget/Flexget/blob/develop/flexget/utils/imdb.py) and then maintain them there. That way, the same list parsing functions could be used by both plugins. I would be happy to help migrating the functions and adapt the imdb_watchlist plugin accordingly, if you decide to do so. However, I don't have time to familiarize myself with the code and adapt the imdb_list plugin to use the shared list parsing functions, that would have to be done by someone else. Also, I suspect that the list modification functions of imdb_list may need updating as well.

If you decide to stop supporting imdb_list completely, it might make sense to update the red deprecation warning on the documentation page saying that imdb_list is not maintained any more and will be removed on (date) unless a maintainer is found. 
iirc, the list is exported to CSV in imdb_list, so it won't make sense to extract the parsing from imdb_watchlist.
@cvium In principle, both plugins need to do the same thing so whether the data is grabbed via a json api, scraped from html or exported via CSV is not so important for the function of the plugin.

Whichever way we use to grab the data, that way is hopefully applicable to both plugins (unless it requires a login, which only imdb_list tries to provide). I just checked and it is possible to export public lists as csv without login.

That being said, imdb_watchlist currently works. I currently don't have time for major changes there. I will have a look into using the CSV export the next time imdb_watchlist breaks. Hopefully, I will be able to unify the data grabbing between imdb_list and imdb_watchlist on that occasion (unless you decide to bury imdb_list).
The issue was about not being able to parse the CSV though, so the user did get the data. This is not a login issue. Some users do get the captcha though and that is a separate issue.
Without a working login it is difficult to debug, though :)

Resolving patch:
diff --git a/flexget/plugins/list/imdb_list.py b/flexget/plugins/list/imdb_list.py
--- a/flexget/plugins/list/imdb_list.py
+++ b/flexget/plugins/list/imdb_list.py
@@ -9,7 +9,7 @@
 from datetime import datetime
 
 from requests.exceptions import RequestException
-
+from requests.utils import cookiejar_from_dict
 from sqlalchemy import Column, Unicode, String
 from sqlalchemy.orm import relation
 from sqlalchemy.schema import ForeignKey
@@ -28,9 +28,9 @@
 
 Base = db_schema.versioned_base('imdb_list', 0)
 
-MOVIE_TYPES = ['feature film', 'documentary', 'tv movie', 'video', 'short film']
-SERIES_TYPES = ['tv series', 'mini-series']
-OTHER_TYPES = ['video game']
+MOVIE_TYPES = ['documentary', 'tvmovie', 'video', 'short', 'movie']
+SERIES_TYPES = ['tvseries', 'tvepisode', 'tvminiseries']
+OTHER_TYPES = ['videogame']
 
 
 class IMDBListUser(Base):
@@ -105,6 +105,18 @@ def session(self):
             self.authenticate()
         return self._session
 
+    def get_user_id(self, cookies=None):
+        try:
+            if cookies:
+                self._session.cookies = cookiejar_from_dict(cookies)
+            # We need to allow for redirects here as it performs 1-2 redirects before reaching the real profile url
+            response = self._session.head('https://www.imdb.com/profile', allow_redirects=True)
+        except RequestException as e:
+            raise PluginError(str(e))
+
+        match = re.search('ur\d+(?!\d)', response.url)
+        return match.group() if match else None
+
     def authenticate(self):
         """Authenticates a session with IMDB, and grabs any IDs needed for getting/modifying list."""
         cached_credentials = False
@@ -112,12 +124,11 @@ def authenticate(self):
             user = session.query(IMDBListUser).filter(IMDBListUser.user_name == self.config.get('login')).one_or_none()
             if user and user.cookies and user.user_id:
                 log.debug('login credentials found in cache, testing')
-                self.cookies = user.cookies
                 self.user_id = user.user_id
-                r = self._session.head('http://www.imdb.com/profile', allow_redirects=False, cookies=self.cookies)
-                if not r.headers.get('location') or 'login' in r.headers['location']:
+                if not self.get_user_id(cookies=user.cookies):
                     log.debug('cache credentials expired')
                 else:
+                    self.cookies = user.cookies
                     cached_credentials = True
             if not cached_credentials:
                 log.debug('user credentials not found in cache or outdated, fetching from IMDB')
@@ -129,25 +140,27 @@ def authenticate(self):
                     'nid.net%2Fauth%2F2.0'
                 )
                 try:
+                    # we need to get some cookies first
+                    self._session.get('https://www.imdb.com')
                     r = self._session.get(url_credentials)
                 except RequestException as e:
                     raise PluginError(e.args[0])
                 soup = get_soup(r.content)
-                inputs = soup.select('form#ap_signin_form input')
+                form = soup.find('form', attrs={'name': 'signIn'})
+                inputs = form.select('input')
                 data = dict((i['name'], i.get('value')) for i in inputs if i.get('name'))
                 data['email'] = self.config['login']
                 data['password'] = self.config['password']
-                action = soup.find('form', id='ap_signin_form').get('action')
+                action = form.get('action')
                 log.debug('email=%s, password=%s', data['email'], data['password'])
                 self._session.headers.update({'Referer': url_credentials})
-                d = self._session.post(action, data=data)
+                self._session.post(action, data=data)
                 self._session.headers.update({'Referer': 'http://www.imdb.com/'})
-                # Get user id by extracting from redirect url
-                r = self._session.head('http://www.imdb.com/profile', allow_redirects=False)
-                if not r.headers.get('location') or 'login' in r.headers['location']:
+
+                self.user_id = self.get_user_id()
+                if not self.user_id:
                     raise plugin.PluginError('Login to IMDB failed. Check your credentials.')
-                self.user_id = re.search('ur\d+(?!\d)', r.headers['location']).group()
-                self.cookies = dict(d.cookies)
+                self.cookies = self._session.cookies.get_dict(domain='.imdb.com')
                 # Get list ID
             if user:
                 for list in user.lists:
@@ -204,28 +217,27 @@ def items(self):
             self._items = []
             for row in csv_reader(lines):
                 log.debug('parsing line from csv: %s', ', '.join(row))
-                if not len(row) == 16:
+                if not len(row) == 15:
                     log.debug('no movie row detected, skipping. %s', ', '.join(row))
                     continue
                 entry = Entry({
-                    'title': '%s (%s)' % (row[5], row[11]) if row[11] != '????' else '%s' % row[5],
-                    'url': row[15],
+                    'title': '%s (%s)' % (row[5], row[10]) if row[10] != '????' else '%s' % row[5],
+                    'url': row[6],
                     'imdb_id': row[1],
-                    'imdb_url': row[15],
+                    'imdb_url': row[6],
                     'imdb_list_position': int(row[0]),
-                    'imdb_list_created': datetime.strptime(row[2], '%a %b %d %H:%M:%S %Y') if row[2] else None,
-                    'imdb_list_modified': datetime.strptime(row[3], '%a %b %d %H:%M:%S %Y') if row[3] else None,
+                    'imdb_list_created': datetime.strptime(row[2], '%Y-%m-%d') if row[2] else None,
+                    'imdb_list_modified': datetime.strptime(row[3], '%Y-%m-%d') if row[3] else None,
                     'imdb_list_description': row[4],
                     'imdb_name': row[5],
-                    'imdb_year': int(row[11]) if row[11] != '????' else None,
-                    'imdb_score': float(row[9]) if row[9] else None,
+                    'imdb_year': int(row[10]) if row[10] != '????' else None,
                     'imdb_user_score': float(row[8]) if row[8] else None,
-                    'imdb_votes': int(row[13]) if row[13] else None,
-                    'imdb_genres': [genre.strip() for genre in row[12].split(',')]
+                    'imdb_votes': int(row[12]) if row[12] else None,
+                    'imdb_genres': [genre.strip() for genre in row[11].split(',')]
                 })
-                item_type = row[6].lower()
+                item_type = row[7].lower()
                 name = row[5]
-                year = int(row[11]) if row[11] != '????' else None
+                year = int(row[10]) if row[10] != '????' else None
                 if item_type in MOVIE_TYPES:
                     entry['movie_name'] = name
                     entry['movie_year'] = year


Test patch: