Repository: pymedusa/Medusa

Issue:
[APP SUBMITTED]: UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 1738: invalid start byte
### INFO
**Python Version**: `2.7.13 (default, Dec 19 2016, 01:28:10) [GCC 4.6.3]`
**Operating System**: `Linux-3.2.40-i686-with-glibc2.0`
**Locale**: `UTF-8`
**Branch**: [master](../tree/master)
**Database**: `44.8`
**Commit**: pymedusa/Medusa@3f23fd272652b10cf781e3058fee50daf79f11b6
**Link to Log**: https://gist.github.com/9e0732da7418a555d443793180809507
### ERROR
<pre>
2017-09-10 12:18:10 ERROR    SNATCHQUEUE-MANUALSNATCH-73141 :: [3f23fd2] DownloadStation: Unknown exception raised when send torrent to DownloadStation : 'utf8' codec can't decode byte 0xc0 in position 1738: invalid start byte
Traceback (most recent call last):
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/medusa/clients/torrent/generic.py#L79">SickBeard/medusa/clients/torrent/generic.py</a>", line 79, in _request
    cookies=cookies, timeout=120, verify=False)
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/lib/requests/sessions.py#L518">SickBeard/lib/requests/sessions.py</a>", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/lib/requests/sessions.py#L475">SickBeard/lib/requests/sessions.py</a>", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/lib/requests/sessions.py#L591">SickBeard/lib/requests/sessions.py</a>", line 591, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/lib/requests/hooks.py#L31">SickBeard/lib/requests/hooks.py</a>", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/<a href="../blob/3f23fd272652b10cf781e3058fee50daf79f11b6/SickBeard/medusa/session/hooks.py#L28">SickBeard/medusa/session/hooks.py</a>", line 28, in log_url
    log.debug('With post data: {data}'.format(data=resp.request.body))
  File "/usr/local/sickbeard-custom/env/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc0 in position 1738: invalid start byte
</pre>
---
_STAFF NOTIFIED_: @pymedusa/support @pymedusa/moderators


Discussion:
This is similar to #3004 however in this case the byte string is not being double mis-converted, its just being decoded with the wrong codec.  It is most likely `ISO 8859-1` which is the codec for West European languages (aka `Latin-1`).  We don't see this issue frequently so I propose we delay this until after #3089 is merged.

The difference between this and #3089 is that for me this is a full stopping error, the torrent is not sent to DownloadStation at all.
Would you mind switching to develop to see if the issue persists or worsens?
So, after finally getting the time to try and switch to develop and also managing to do that, the issue still persists unfortunately. Same error message as in original post.
Oops, closed the issue by mistake (close and comment did not mean close this form and go back to the list 😄  )
@andersjohansson3 any chance you have Latin-1 characters (e.g. Ā, Ē, etc) in your username or password?
@labrys No I don't have any Latin-1 in either Medusa, Synology or the providers user/pass
Ok.  If you could update and try the latest develop branch.  It should fix your issue.
Tried and unfortunately it's the same issue still. Debug log below:

```
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Transaction with 1 queries executed
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] 278518: Formatting pattern: %SN - %Sx%0E - %EN - %QN -> Last Week Tonight with John Oliver - 4x23 - Episode 112 - 1080p WEB-DL
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Found codec for Last Week Tonight with John Oliver Episode 112
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Checking if we should stop searching for a better quality for for episode Last.Week.Tonight.with.John.Oliver.S04E23.1080p.AMZN.WEB-DL.DD+2.0.H.264-monkee
2017-09-17 18:18:15 ERROR    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] DownloadStation: Unknown exception raised when send torrent to DownloadStation : 'utf8' codec can't decode byte 0x8b in position 1614: invalid start byte
Traceback (most recent call last):
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/medusa/clients/torrent/generic.py", line 79, in _request
    cookies=cookies, timeout=120, verify=False)
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/lib/requests/sessions.py", line 518, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/lib/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/lib/requests/sessions.py", line 591, in send
    r = dispatch_hook('response', hooks, r, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/lib/requests/hooks.py", line 31, in dispatch_hook
    _hook_data = hook(hook_data, **kwargs)
  File "/volume1/@appstore/sickbeard-custom/var/SickBeard/medusa/session/hooks.py", line 28, in log_url
    data = resp.request.body.decode('utf-8')
  File "/usr/local/sickbeard-custom/env/lib/python2.7/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x8b in position 1614: invalid start byte
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] User-Agent: Medusa/0.0.1(Linux; 3.2.40; 45cb889c-9bc2-11e7-b072-001132374582)
2017-09-17 18:18:15 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] POST URL: http://localhost:9092/webapi/DownloadStation/task.cgi [Status: 200]
2017-09-17 18:18:14 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] DownloadStation: Requested a POST connection to http://localhost:9092/webapi/DownloadStation/task.cgi with params: None Data: {u'session': u'DownloadStation', u'destination': u'/volume1/downloads/downloads_tv', u'api': u'SYNO...
2017-09-17 18:18:14 INFO     SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Destination is now downloads/downloads_tv
2017-09-17 18:18:14 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] User-Agent: Medusa/0.0.1(Linux; 3.2.40; 45cb889c-9bc2-11e7-b072-001132374582)
2017-09-17 18:18:14 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] GET URL: http://localhost:9092/webapi/entry.cgi?session=DownloadStation&api=SYNO.DSM.Info&version=2&method=getinfo [Status: 200]
2017-09-17 18:18:14 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] User-Agent: Medusa/0.0.1(Linux; 3.2.40; 45cb889c-9bc2-11e7-b072-001132374582)
2017-09-17 18:18:14 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] GET URL: http://localhost:9092/webapi/auth.cgi?session=DownloadStation&account=**********&version=2&format=cookie&passwd=**********&api=SYNO.API.Auth&method=login [Status: 200]
2017-09-17 18:18:13 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Calling DownloadStation Client
2017-09-17 18:18:13 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] User-Agent: Mozilla/5.0 (Windows NT 10.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/40.0.2214.93 Safari/537.36
2017-09-17 18:18:13 DEBUG    SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] GET URL: https://iptorrents.eu/download.php/2464638/Last.Week.Tonight.with.John.Oliver.S04E23.1080p.AMZN.WEB-DL.DD+2.0.H.264-monkee.torrent [Status: 200]
2017-09-17 18:18:13 INFO     SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Downloading Last.Week.Tonight.with.John.Oliver.S04E23.1080p.AMZN.WEB-DL.DD+2.0.H.264-monkee with 92 seeders and 3 leechers and size 1.63 GB from IPTorrents
2017-09-17 18:18:13 INFO     SNATCHQUEUE-MANUALSNATCH-278518 :: [bc0a5ce] Beginning to manual snatch release: Last.Week.Tonight.with.John.Oliver.S04E23.1080p.AMZN.WEB-DL.DD+2.0.H.264-monkee
```


Resolving patch:
diff --git a/medusa/session/hooks.py b/medusa/session/hooks.py
--- a/medusa/session/hooks.py
+++ b/medusa/session/hooks.py
@@ -13,22 +13,34 @@
 log.addHandler(logging.NullHandler())
 
 
-def log_url(resp, **kwargs):
+def log_url(response, **kwargs):
     """Response hook to log request URL."""
+    request = response.request
     log.debug(
         '{method} URL: {url} [Status: {status}]'.format(
-            method=resp.request.method,
-            url=resp.request.url,
-            status=resp.status_code,
+            method=request.method,
+            url=request.url,
+            status=response.status_code,
         )
     )
-    log.debug('User-Agent: {}'.format(resp.request.headers['User-Agent']))
-
-    if resp.request.method.upper() == 'POST':
-        data = resp.request.body.decode('utf-8')
-        log.debug('With post data: {data}'.format(data=data))
-
-    return resp
+    log.debug('User-Agent: {}'.format(request.headers['User-Agent']))
+
+    if request.method.upper() == 'POST':
+        body = request.body
+        # try to log post data using various codecs to decode
+        codecs = ('utf-8', 'latin1', 'cp1252')
+        for codec in codecs:
+            try:
+                data = body.decode(codec)
+            except UnicodeError as error:
+                log.debug('Failed to decode post data as {codec}: {msg}',
+                          codec=codec, msg=error)
+            else:
+                log.debug('With post data: {0}', data)
+                break
+        else:
+            log.warning('Failed to decode post data with {codecs}',
+                        codecs=codecs)
 
 
 def cloudflare(resp, **kwargs):


Test patch: