Repository: Flexget/Flexget

Issue:
IMDb login fails
### Expected behaviour:

imdb_list plugin logins in successfully with the logged in credentials.
### Actual behaviour:

Login fails.
### Steps to reproduce:
- Step 1: excute configuration
#### Config:

```
tasks:
  testtask:
    rss: <feed>
    imdb_lookup: yes
    list_match:
      from:
        - imdb_list:
            login: <username>
            password: <password
            list: watchlist
    download: <dir>
```
#### Log:

```
2016-08-08 20:26 VERBOSE utils.requests testtask Waiting 3.83 seconds until next request to imdb.com
2016-08-08 20:26 VERBOSE utils.requests testtask Waiting 4.66 seconds until next request to imdb.com
2016-08-08 20:26 CRITICAL plugin testtask Login to IMDB failed. Check your credentials.
2016-08-08 20:26 WARNING task testtask Aborting task (plugin: list_match)
```
### Additional information:

liiight advies to raise a ticket. Test was done by Krozer.
- Flexget Version: 2.2.13
- Python Version: 2.7.11
- Installation method: pip
- OS and version: Mac OS X 10.11.6
- Link to crash log:
### Test done by Krozer:

```
import re
import pprint

from flexget.utils.requests import Session as RequestSession, TimedLimiter
from requests.exceptions import RequestException
from flexget.utils.soup import get_soup
from flexget.plugin import PluginError


pp = pprint.PrettyPrinter(indent=4)
_session = RequestSession()
_session.add_domain_limiter(TimedLimiter('imdb.com', '5 seconds'))
_session.headers = {'Accept-Language': 'en-us'}

try:
    r = _session.get(
        'https://www.imdb.com/ap/signin?openid.return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-'
        'handler&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&'
        'openid.assoc_handle=imdb_mobile_us&openid.mode=checkid_setup&openid.claimed_id=http%3A%'
        '2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.ope'
        'nid.net%2Fauth%2F2.0')
except RequestException as e:
    raise PluginError(e.args[0])

soup = get_soup(r.content)
inputs = soup.select('form#ap_signin_form input')
data = dict((i['name'], i.get('value')) for i in inputs if i.get('name'))
action = soup.find('form', id='ap_signin_form').get('action')
pp.pprint(action)


data['email'] = <email>
data['password'] = <password>
print('email=%s, password=%s', data['email'], data['password'])


d = _session.post(action, data=data)
# Get user id by extracting from redirect url
r = _session.head('http://www.imdb.com/profile', allow_redirects=False)
pp.pprint(r.headers)
if not r.headers.get('location') or 'login' in r.headers['location']:
    raise PluginError('Login to IMDB failed. Check your credentials.')
user_id = re.search('ur\d+(?!\d)', r.headers['location']).group()
cookies = dict(d.cookies)
```



Discussion:
Hello,

After some research, it seems that the problem is in the header sent.

This method overwrite the header.:
`_session.headers = { 'Accept-Language': 'en-us'}`

From what I understood to be used:
`_session.headers.update ({  'Accept-Language': 'en-us' })`

The header sent can be seen with this:

```
r.request.headers.keys ()
   print (key, r.request.headers [key])
```

for my tests I used this header after _session statement:

```
_session.add_domain_limiter(TimedLimiter('imdb.com', '5 seconds'))
_session.headers.update({'Accept-Encoding': 'deflate'})
_session.headers.update({'Accept-Language':'fr-FR,fr;q=0.8,en-US;q=0.6,en;q=0.4'})
_session.headers.update({'Cache-Control':'max-age=0'})
_session.headers.update({'Host':'www.imdb.com'})
_session.headers.update({'Upgrade-Insecure-Requests':'1'})
_session.headers.update({'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36'})
```

And add this before the post request :

```
_session.headers.update({'Origin':'https://www.imdb.com'})
_session.headers.update({'Referer':'https://www.imdb.com/ap/signin?openid.return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-'
    'handler&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&'
    'openid.assoc_handle=imdb_mobile_us&openid.mode=checkid_setup&openid.claimed_id=http%3A%'
    '2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.ope'
    'nid.net%2Fauth%2F2.0'})
```

Now I log properly but if there are too many connection in some time then imdb activates a system of capcha.

It would look to cookiesJar to save cookies to a file to avoid this problem.

ps: This is my first time with python, sorry it will not be the pretty code, and sorry for my poor English

You are correct that it should update headers rather than assign but I doubt that this is the issue because it has been working with this from the start. 
It does appear however that imdb changed the watchlist : https://getsatisfaction.com/imdb/topics/new-version-of-imdb-watchlist-launched
We're still investigating this though. 


Resolving patch:
diff --git a/flexget/plugins/list/imdb_list.py b/flexget/plugins/list/imdb_list.py
--- a/flexget/plugins/list/imdb_list.py
+++ b/flexget/plugins/list/imdb_list.py
@@ -88,7 +88,7 @@ def __init__(self, config):
         self.config = config
         self._session = RequestSession()
         self._session.add_domain_limiter(TimedLimiter('imdb.com', '5 seconds'))
-        self._session.headers = {'Accept-Language': config.get('force_language', 'en-us')}
+        self._session.headers.update( {'Accept-Language': config.get('force_language', 'en-us')} )
         self.user_id = None
         self.list_id = None
         self.cookies = None
@@ -117,13 +117,14 @@ def authenticate(self):
                     cached_credentials = True
             if not cached_credentials:
                 log.debug('user credentials not found in cache or outdated, fetching from IMDB')
+                url_credentials = ( 'https://www.imdb.com/ap/signin?openid.return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-'
+                    'handler&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&'
+                    'openid.assoc_handle=imdb_mobile_us&openid.mode=checkid_setup&openid.claimed_id=http%3A%'
+                    '2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.ope'
+                    'nid.net%2Fauth%2F2.0'
+                )
                 try:
-                    r = self._session.get(
-                        'https://www.imdb.com/ap/signin?openid.return_to=https%3A%2F%2Fwww.imdb.com%2Fap-signin-'
-                        'handler&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&'
-                        'openid.assoc_handle=imdb_mobile_us&openid.mode=checkid_setup&openid.claimed_id=http%3A%'
-                        '2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.ns=http%3A%2F%2Fspecs.ope'
-                        'nid.net%2Fauth%2F2.0')
+                    r = self._session.get(url_credentials)
                 except RequestException as e:
                     raise PluginError(e.args[0])
                 soup = get_soup(r.content)
@@ -131,8 +132,11 @@ def authenticate(self):
                 data = dict((i['name'], i.get('value')) for i in inputs if i.get('name'))
                 data['email'] = self.config['login']
                 data['password'] = self.config['password']
+                action = soup.find('form', id='ap_signin_form').get('action')
                 log.debug('email=%s, password=%s', data['email'], data['password'])
-                d = self._session.post('https://www.imdb.com/ap/signin', data=data)
+                self._session.headers.update({'Referer':url_credentials})
+                d = self._session.post(action, data=data)
+                self._session.headers.update({'Referer':'http://www.imdb.com/'})
                 # Get user id by extracting from redirect url
                 r = self._session.head('http://www.imdb.com/profile', allow_redirects=False)
                 if not r.headers.get('location') or 'login' in r.headers['location']:


Test patch:
diff --git a/flexget/tests/cassettes/test_imdb_list_interface.TestIMDBListTypes.test_imdb_list b/flexget/tests/cassettes/test_imdb_list_interface.TestIMDBListTypes.test_imdb_list
--- a/flexget/tests/cassettes/test_imdb_list_interface.TestIMDBListTypes.test_imdb_list
+++ b/flexget/tests/cassettes/test_imdb_list_interface.TestIMDBListTypes.test_imdb_list
@@ -264,7 +264,7 @@ interactions:
       Content-Type: [application/x-www-form-urlencoded]
       Cookie: [session-id=166-1066870-6775531; session-id-time=2100087888l]
     method: POST
-    uri: https://www.imdb.com/ap/signin
+    uri: https://www.imdb.com/ap/signin/166-1066870-6775531?ie=UTF8&pf_rd_r=1P284FWD1YWTTE42RGG8&pf_rd_m=A2A1PIL045UJ9G&pf_rd_t=6301&pf_rd_i=imdb_mobile_us&pf_rd_p=2187562082&pf_rd_s=mobile-signin-slot
   response:
     body: {string: !!python/unicode ''}
     headers: