Repository: Flexget/Flexget

Issue:
Failed by payload plugin because 91
2016-05-04 17:43 VERBOSE  task **\* FAILED: `***` by pyload plugin because 91

Failed by payload plugin because 91
2016-05-04 17:43 VERBOSE  task **\* FAILED: `***` by pyload plugin because 91



Discussion:
congratulations, i think this is the worst bug description i ever saw in Flexget ;-)
please paste full config and log

Well actually I didn't want to post it right away, just a reminder for myself :P
But now it is here ...

```
input_serienjunkies_de_ul:
    rss: http://serienjunkies.org/xml/feeds/episoden.xml
    serienjunkies:
      language: german
      hoster: ul

  download_pyload:
    pyload:
      api: http://192.168.22.10:8000/api
      queue: yes
      package: "{{series_name|replace(\"'\", \"\")}} - {{series_id}} - {{quality}}"
      package_password: ***
      username: ***
      password: ***`
```

Everything else is basically task scheduling and other stuff, loading the rss, parsing, filtering, url-rewriting all working fine, but sending the download-links to payload throws this error, just updated to v2, worked fine before

you forgot to add full log, debug is best

[verbose_log.txt](https://github.com/Flexget/Flexget/files/248880/verbose_log.txt)

reset retry_failed and run again please, we need the trace:
`flexget failed clear`

2016-05-04 18:17 CRITICAL payload **\* Unknown error: 'bool' object has no attribute 'replace'
[verbose_log.txt](https://github.com/Flexget/Flexget/files/248895/verbose_log.txt)

i think that's the same file. also, please use a paste service like http://hastebin.com/

It is not, but you are right, only difference is the copied line above ...
http://pastebin.com/za4ff7KV

Seems to be fixed with a reinstall with python3

I have done a reinstall of flexget and I am still seeing the same issue

Same for me. I also did a fresh install of pyload.

Using Python 3 or Python 2? My first try was with Python 2 and reinstall on a clean Python 3!

I'm not certain if it's related to the problem. But the config schema for pyload plugin looks a bit wrong. According to the schema `pyload: yes` is a valid config, but it looks like that will crash. Also, in dict form of config, none of the fields are marked as required, which I think they probably should be.

@kasperj So you uninstalled python 2 and flexget and than installed python 3 with flexget?

I was migrating to a new device so its a brand new debian vm with python 3 and flexget

python 2 for me, i have not updated anything other than flexget.

I rolled back to version 1.2.521 and everything is working correctly again

@jbman11 How did u installed the old version of flexget?

sudo pip install flexget==1.2.521

How can we get this reopened, since the fix to install on a clean system with py3 isn't an option for all?

I reopened the issue

Anything needed to help get this one looked at?

With Python3 I'm getting this error in Pyload:

`ValueError: 'b'Althaus Modernisieren - August/September 2016'' is not a valid expression`

When you look at [this line](https://github.com/Flexget/Flexget/blob/master/flexget/plugins/output/pyload.py#L195) you see that the title is encoded to bytes and then inserted into a string, which explains the weird value passed to Pyload.

I understand that this is a trick to remove weird characters and put quotes around them. The solution then is to add another `.decode()` to turn `bytes` back into a string and avoids passing those single quotes to Pyload.

I can do a proper PR and test it under py2 and py3, if there is still need.

It would be great if this could be fixed. I had to roll back to 1.2.521 and stay there

@m3nu Any chance you can explain what exact change needs to be made regarding this line? I am receiving the exact same error with Python 3.

On another note, with Python27 I also receive error 91

Just make sure pyload gets a valid title. I simply added another `decode()` at the end.

```
post = {'name': "'%s'" % name.encode("ascii", "ignore").decode()
```

So this line encodes to ascii bytes (to remove crazy characters) and then back to a string, so pyload doesn't get the byte representationt with extra quotes.

Still facing this problem in Python 2.7. Tried the fix from previous post with no luck (flexget -V = 2.3.16 & pyLoad is 4.9.)

> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/login
> 2016-09-06 13:23 DEBUG    pyload        get_magazines_3 Parsing url "http://somewebsite.com/pickedpage.html"
> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/parseURLs
> 2016-09-06 13:23 DEBUG    pyload        get_magazines_3 Add 1 urls to pyLoad
> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/addPackage
> 2016-09-06 13:23 DEBUG    entry         get_magazines_3 Marking entry 'One Magazine- September 2016' as failed
> 2016-09-06 13:23 ERROR    entry         get_magazines_3 Failed One Magazine - September 2016 (91)

This is my TASK:

```
  get_magazines_3:
    priority: 24
    html: http://somewebsite.com/keyword/
    regexp:
      accept:
       - One Magazine
      rest: reject
    pyload:
      api: http://192.168.1.4:8000/api
      username: *********
      password: **********
#      package: "{{url}}"
      queue: yes
      parse_url: yes
      hoster: 
        - UploadedTo
      multiple_hoster: no
      enabled: yes
```

See nothing on pyload log side. Is there a way to make it more verbose?

@m3nu Thanks, that fixed it for me. I'd say submit PR, but I don't know the implications for Python 2.7.

congratulations, i think this is the worst bug description i ever saw in Flexget ;-)
please paste full config and log

Well actually I didn't want to post it right away, just a reminder for myself :P
But now it is here ...

```
input_serienjunkies_de_ul:
    rss: http://serienjunkies.org/xml/feeds/episoden.xml
    serienjunkies:
      language: german
      hoster: ul

  download_pyload:
    pyload:
      api: http://192.168.22.10:8000/api
      queue: yes
      package: "{{series_name|replace(\"'\", \"\")}} - {{series_id}} - {{quality}}"
      package_password: ***
      username: ***
      password: ***`
```

Everything else is basically task scheduling and other stuff, loading the rss, parsing, filtering, url-rewriting all working fine, but sending the download-links to payload throws this error, just updated to v2, worked fine before

you forgot to add full log, debug is best

[verbose_log.txt](https://github.com/Flexget/Flexget/files/248880/verbose_log.txt)

reset retry_failed and run again please, we need the trace:
`flexget failed clear`

2016-05-04 18:17 CRITICAL payload **\* Unknown error: 'bool' object has no attribute 'replace'
[verbose_log.txt](https://github.com/Flexget/Flexget/files/248895/verbose_log.txt)

i think that's the same file. also, please use a paste service like http://hastebin.com/

It is not, but you are right, only difference is the copied line above ...
http://pastebin.com/za4ff7KV

Seems to be fixed with a reinstall with python3

I have done a reinstall of flexget and I am still seeing the same issue

Same for me. I also did a fresh install of pyload.

Using Python 3 or Python 2? My first try was with Python 2 and reinstall on a clean Python 3!

I'm not certain if it's related to the problem. But the config schema for pyload plugin looks a bit wrong. According to the schema `pyload: yes` is a valid config, but it looks like that will crash. Also, in dict form of config, none of the fields are marked as required, which I think they probably should be.

@kasperj So you uninstalled python 2 and flexget and than installed python 3 with flexget?

I was migrating to a new device so its a brand new debian vm with python 3 and flexget

python 2 for me, i have not updated anything other than flexget.

I rolled back to version 1.2.521 and everything is working correctly again

@jbman11 How did u installed the old version of flexget?

sudo pip install flexget==1.2.521

How can we get this reopened, since the fix to install on a clean system with py3 isn't an option for all?

I reopened the issue

Anything needed to help get this one looked at?

With Python3 I'm getting this error in Pyload:

`ValueError: 'b'Althaus Modernisieren - August/September 2016'' is not a valid expression`

When you look at [this line](https://github.com/Flexget/Flexget/blob/master/flexget/plugins/output/pyload.py#L195) you see that the title is encoded to bytes and then inserted into a string, which explains the weird value passed to Pyload.

I understand that this is a trick to remove weird characters and put quotes around them. The solution then is to add another `.decode()` to turn `bytes` back into a string and avoids passing those single quotes to Pyload.

I can do a proper PR and test it under py2 and py3, if there is still need.

It would be great if this could be fixed. I had to roll back to 1.2.521 and stay there

@m3nu Any chance you can explain what exact change needs to be made regarding this line? I am receiving the exact same error with Python 3.

On another note, with Python27 I also receive error 91

Just make sure pyload gets a valid title. I simply added another `decode()` at the end.

```
post = {'name': "'%s'" % name.encode("ascii", "ignore").decode()
```

So this line encodes to ascii bytes (to remove crazy characters) and then back to a string, so pyload doesn't get the byte representationt with extra quotes.

Still facing this problem in Python 2.7. Tried the fix from previous post with no luck (flexget -V = 2.3.16 & pyLoad is 4.9.)

> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/login
> 2016-09-06 13:23 DEBUG    pyload        get_magazines_3 Parsing url "http://somewebsite.com/pickedpage.html"
> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/parseURLs
> 2016-09-06 13:23 DEBUG    pyload        get_magazines_3 Add 1 urls to pyLoad
> 2016-09-06 13:23 DEBUG    utils.requests get_magazines_3 Fetching http://192.168.1.4:8000/api/addPackage
> 2016-09-06 13:23 DEBUG    entry         get_magazines_3 Marking entry 'One Magazine- September 2016' as failed
> 2016-09-06 13:23 ERROR    entry         get_magazines_3 Failed One Magazine - September 2016 (91)

This is my TASK:

```
  get_magazines_3:
    priority: 24
    html: http://somewebsite.com/keyword/
    regexp:
      accept:
       - One Magazine
      rest: reject
    pyload:
      api: http://192.168.1.4:8000/api
      username: *********
      password: **********
#      package: "{{url}}"
      queue: yes
      parse_url: yes
      hoster: 
        - UploadedTo
      multiple_hoster: no
      enabled: yes
```

See nothing on pyload log side. Is there a way to make it more verbose?

@m3nu Thanks, that fixed it for me. I'd say submit PR, but I don't know the implications for Python 2.7.


Resolving patch:
diff --git a/flexget/plugins/output/pyload.py b/flexget/plugins/output/pyload.py
--- a/flexget/plugins/output/pyload.py
+++ b/flexget/plugins/output/pyload.py
@@ -23,24 +23,27 @@ def __init__(self, requests, url):
 
     def get_session(self, config):
         # Login
-        post = {'username': config['username'], 'password': config['password']}
-        result = self.query("login", post)
+        data = {'username': config['username'], 'password': config['password']}
+        result = self.post('login', data=data)
         response = result.json()
         if not response:
             raise plugin.PluginError('Login failed', log)
         return response.replace('"', '')
 
-    def query(self, method, post=None):
+    def get(self, method):
         try:
-            response = self.requests.request(
-                'post' if post is not None else 'get',
-                self.url.rstrip("/") + "/" + method.strip("/"),
-                data=post)
-            response.raise_for_status()
-            return response
+            return self.requests.get(self.url.rstrip("/") + "/" + method.strip("/"))
         except RequestException as e:
             if e.response and e.response.status_code == 500:
-                raise plugin.PluginError('Internal API Error: <%s> <%s> <%s>' % (method, self.url, post), log)
+                raise plugin.PluginError('Internal API Error: <%s> <%s>' % (method, self.url), log)
+            raise
+
+    def post(self, method, data):
+        try:
+            return self.requests.post(self.url.rstrip("/") + "/" + method.strip("/"), data=data)
+        except RequestException as e:
+            if e.response and e.response.status_code == 500:
+                raise plugin.PluginError('Internal API Error: <%s> <%s> <%s>' % (method, self.url, data), log)
             raise
 
 
@@ -88,28 +91,24 @@ class PluginPyLoad(object):
     DEFAULT_HANDLE_NO_URL_AS_FAILURE = False
 
     schema = {
-        'oneOf': [
-            {'type': 'boolean'},
-            {'type': 'object',
-             'properties': {
-                 'api': {'type': 'string'},
-                 'username': {'type': 'string'},
-                 'password': {'type': 'string'},
-                 'folder': {'type': 'string'},
-                 'package': {'type': 'string'},
-                 'package_password': {'type': 'string'},
-                 'queue': {'type': 'boolean'},
-                 'parse_url': {'type': 'boolean'},
-                 'multiple_hoster': {'type': 'boolean'},
-                 'hoster': one_or_more({'type': 'string'}),
-                 'preferred_hoster_only': {'type': 'boolean'},
-                 'handle_no_url_as_failure': {'type': 'boolean'},
-                 'enabled': {'type': 'boolean'},
-
-             },
-             'additionalProperties': False
-             }
-        ]
+        'type': 'object',
+        'properties': {
+            'api': {'type': 'string'},
+            'username': {'type': 'string'},
+            'password': {'type': 'string'},
+            'folder': {'type': 'string'},
+            'package': {'type': 'string'},
+            'package_password': {'type': 'string'},
+            'queue': {'type': 'boolean'},
+            'parse_url': {'type': 'boolean'},
+            'multiple_hoster': {'type': 'boolean'},
+            'hoster': one_or_more({'type': 'string'}),
+            'preferred_hoster_only': {'type': 'boolean'},
+            'handle_no_url_as_failure': {'type': 'boolean'},
+            'enabled': {'type': 'boolean'}
+        },
+        'required': ['username', 'password'],
+        'additionalProperties': False
     }
 
     def on_task_output(self, task, config):
@@ -144,11 +143,11 @@ def add_entries(self, task, config):
 
             url = json.dumps(entry['url']) if config.get('parse_url', self.DEFAULT_PARSE_URL) else "''"
 
-            log.debug("Parsing url %s" % url)
+            log.debug('Parsing url %s', url)
 
-            result = api.query("parseURLs", {"html": content, "url": url, "session": session})
+            data = {'html': content, 'url': url, 'session': session}
+            result = api.post('parseURLs', data=data)
 
-            # parsed { plugins: [urls] }
             parsed = result.json()
 
             urls = []
@@ -163,22 +162,22 @@ def add_entries(self, task, config):
             # no preferred hoster and not preferred hoster only - add all recognized plugins
             if not urls and not config.get('preferred_hoster_only', self.DEFAULT_PREFERRED_HOSTER_ONLY):
                 for name, purls in parsed.items():
-                    if name != "BasePlugin":
+                    if name != 'BasePlugin':
                         urls.extend(purls)
 
             if task.options.test:
-                log.info('Would add `%s` to pyload' % urls)
+                log.info('Would add `%s` to pyload', urls)
                 continue
 
             # no urls found
             if not urls:
                 if config.get('handle_no_url_as_failure', self.DEFAULT_HANDLE_NO_URL_AS_FAILURE):
-                    entry.fail("No suited urls in entry %s" % entry['title'])
+                    entry.fail('No suited urls in entry %s' % entry['title'])
                 else:
-                    log.info("No suited urls in entry %s" % entry['title'])
+                    log.info('No suited urls in entry %s', entry['title'])
                 continue
 
-            log.debug("Add %d urls to pyLoad" % len(urls))
+            log.debug('Add %d urls to pyLoad', len(urls))
 
             try:
                 dest = 1 if config.get('queue', self.DEFAULT_QUEUE) else 0  # Destination.Queue = 1
@@ -191,15 +190,15 @@ def add_entries(self, task, config):
                     name = entry.render(name)
                 except RenderError as e:
                     name = entry['title']
-                    log.error('Error rendering jinja event: %s' % e)
+                    log.error('Error rendering jinja event: %s', e)
 
-                post = {'name': "'%s'" % name.encode("ascii", "ignore"),
-                        'links': str(urls),
-                        'dest': dest,
+                data = {'name': json.dumps(name.encode('ascii', 'ignore').decode()),
+                        'links': json.dumps(urls),
+                        'dest': json.dumps(dest),
                         'session': session}
 
-                pid = api.query("addPackage", post).text
-                log.debug('added package pid: %s' % pid)
+                pid = api.post('addPackage', data=data).text
+                log.debug('added package pid: %s', pid)
 
                 # Set Folder
                 folder = config.get('folder', self.DEFAULT_FOLDER)
@@ -210,16 +209,16 @@ def add_entries(self, task, config):
                         folder = entry.render(folder)
                     except RenderError as e:
                         folder = self.DEFAULT_FOLDER
-                        log.error('Error rendering jinja event: %s' % e)
+                        log.error('Error rendering jinja event: %s', e)
                     # set folder with api
                     data = json.dumps({'folder': folder})
-                    api.query("setPackageData", {'pid': pid, 'data': data, 'session': session})
+                    api.post("setPackageData", data={'pid': pid, 'data': data, 'session': session})
 
                 # Set Package Password
                 package_password = config.get('package_password')
                 if package_password:
                     data = json.dumps({'password': package_password})
-                    api.query("setPackageData", {'pid': pid, 'data': data, 'session': session})
+                    api.post('setPackageData', data={'pid': pid, 'data': data, 'session': session})
 
             except Exception as e:
                 entry.fail(str(e))


Test patch: