Repository: scrapy/scrapy

Issue:
PY3: error decoding Content-Disposition header
This request

```
scrapy shell 'http://npe.com.cn/plus/save_to_doc.php?id=1666'
```

raises this error:

```
Traceback (most recent call last):
  File "/Users/kmike/envs/dl/bin/scrapy", line 9, in <module>
    load_entry_point('Scrapy', 'console_scripts', 'scrapy')()
  File "/Users/kmike/svn/scrapy/scrapy/cmdline.py", line 142, in execute
    _run_print_help(parser, _run_command, cmd, args, opts)
  File "/Users/kmike/svn/scrapy/scrapy/cmdline.py", line 88, in _run_print_help
    func(*a, **kw)
  File "/Users/kmike/svn/scrapy/scrapy/cmdline.py", line 149, in _run_command
    cmd.run(args, opts)
  File "/Users/kmike/svn/scrapy/scrapy/commands/shell.py", line 71, in run
    shell.start(url=url)
  File "/Users/kmike/svn/scrapy/scrapy/shell.py", line 47, in start
    self.fetch(url, spider)
  File "/Users/kmike/svn/scrapy/scrapy/shell.py", line 112, in fetch
    reactor, self._schedule, request, spider)
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/python/failure.py", line 368, in raiseException
    raise self.value.with_traceback(self.tb)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 25: invalid start byte
```

The error points to a wrong location (similar to #1760); the real traceback is

```
Traceback (most recent call last):
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 1126, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/Users/kmike/svn/scrapy/scrapy/core/downloader/middleware.py", line 43, in process_request
    defer.returnValue((yield download_func(request=request,spider=spider)))
  File "/Users/kmike/envs/dl/lib/python3.5/site-packages/Twisted-15.5.0-py3.5.egg/twisted/internet/defer.py", line 588, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/Users/kmike/svn/scrapy/scrapy/core/downloader/handlers/http11.py", line 272, in _cb_bodydone
    respcls = responsetypes.from_args(headers=headers, url=url)
  File "/Users/kmike/svn/scrapy/scrapy/responsetypes.py", line 110, in from_args
    cls = self.from_headers(headers)
  File "/Users/kmike/svn/scrapy/scrapy/responsetypes.py", line 78, in from_headers
    cls = self.from_content_disposition(headers[b'Content-Disposition'])
  File "/Users/kmike/svn/scrapy/scrapy/responsetypes.py", line 62, in from_content_disposition
    filename = to_native_str(content_disposition).split(';')[1].split('=')[1]
  File "/Users/kmike/svn/scrapy/scrapy/utils/python.py", line 129, in to_native_str
    return to_unicode(text, encoding, errors)
  File "/Users/kmike/svn/scrapy/scrapy/utils/python.py", line 107, in to_unicode
    return text.decode(encoding, errors)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb8 in position 25: invalid start byte
```

It looks like Content-Disposition is decoded using utf-8, but the encoding was not UTF-8.



Discussion:
The change came in https://github.com/scrapy/scrapy/pull/1394

```
diff --git a/scrapy/responsetypes.py b/scrapy/responsetypes.py
index c212f57..5b0bc50 100644
--- a/scrapy/responsetypes.py
+++ b/scrapy/responsetypes.py
@@ -11,7 +11,7 @@

 from scrapy.http import Response
 from scrapy.utils.misc import load_object
-from scrapy.utils.python import isbinarytext
+from scrapy.utils.python import isbinarytext, to_bytes, to_native_str

 class ResponseTypes(object):

@@ -59,7 +59,7 @@ def from_content_type(self, content_type, content_encoding=None):

     def from_content_disposition(self, content_disposition):
         try:
-            filename = content_disposition.split(';')[1].split('=')[1]
+            filename = to_native_str(content_disposition).split(';')[1].split('=')[1]
             filename = filename.strip('"\'')
             return self.from_filename(filename)
         except IndexError:
```

On Python 2, you get `str`-typed value:

```
(scrapydev)$ python --version
Python 2.7.10
(scrapydev)$ scrapy shell 'http://npe.com.cn/plus/save_to_doc.php?id=1666'
2016-02-17 11:34:36 [scrapy] INFO: Scrapy 1.2.0dev2 started (bot: scrapybot)
...
2016-02-17 11:34:42 [scrapy] DEBUG: Crawled (200) <GET http://npe.com.cn/plus/save_to_doc.php?id=1666> (referer: None)
...
In [1]: response.headers
Out[1]: 
{'Cache-Control': 'no-cache',
 'Content-Disposition': 'attachment;filename=1080P\xb8\xdf\xc7\xe5\xcd\xf8\xc2\xe7\xb7\xc0\xb1\xa9\xb0\xeb\xc7\xf2\xc9\xe3\xcf\xf1\xbb\xfa NCC-5320CV-1.doc',
 'Content-Type': 'text/plain',
 'Date': 'Wed, 17 Feb 2016 10:34:52 GMT',
 'Expires': '0',
 'Pragma': 'public',
 'Server': 'Microsoft-IIS/7.5',
 'Vary': 'Accept-Encoding',
 'X-Powered-By': 'ASP.NET'}

In [2]: type(response.headers['content-disposition'])
Out[2]: str
```

The encoding is probably `gbk`, but I don't see any hint on this apart from the `.com.cn` in the domain name, and even in my Chrome browser, the name of the downloaded file was wrong (latin-1 decoded): 

```
1080P¸ßÇåÍøÂç·À±©°ëÇòÉãÏñ»ú NCC-5320CV-1.doc
http://npe.com.cn/plus/save_to_doc.php?id=1666
```

Decoding tests:

```

In [3]: response.headers['content-disposition'].decode('utf-8')
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-3-0b9c4b5aa4e0> in <module>()
----> 1 response.headers['content-disposition'].decode('utf-8')

/home/paul/.virtualenvs/scrapydev/lib/python2.7/encodings/utf_8.pyc in decode(input, errors)
     14 
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17 
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeDecodeError: 'utf8' codec can't decode byte 0xb8 in position 25: invalid start byte

In [4]: response.headers['content-disposition'].decode('gbk')
Out[4]: u'attachment;filename=1080P\u9ad8\u6e05\u7f51\u7edc\u9632\u66b4\u534a\u7403\u6444\u50cf\u673a NCC-5320CV-1.doc'

In [5]: print(response.headers['content-disposition'].decode('gbk'))
attachment;filename=1080P高清网络防暴半球摄像机 NCC-5320CV-1.doc

In [6]: print(response.headers['content-disposition'].decode('big5'))
attachment;filename=1080P詢ь厙釐滅惟圉⑩扜砉儂 NCC-5320CV-1.doc

In [7]: print(response.headers['content-disposition'].decode('latin-1'))
attachment;filename=1080P¸ßÇåÍøÂç·À±©°ëÇòÉãÏñ»ú NCC-5320CV-1.doc

```

we can probably decode with `'replace'` at least, to limit damage.

Alternatively, we can decode header with latin1 like browsers do. We almost don't care about encoding: this header is used to get a file name; scrapy uses the extension of this file name to guess the content type, and all extensions are ascii-only.


Resolving patch:
diff --git a/scrapy/responsetypes.py b/scrapy/responsetypes.py
--- a/scrapy/responsetypes.py
+++ b/scrapy/responsetypes.py
@@ -59,7 +59,8 @@ def from_content_type(self, content_type, content_encoding=None):
 
     def from_content_disposition(self, content_disposition):
         try:
-            filename = to_native_str(content_disposition).split(';')[1].split('=')[1]
+            filename = to_native_str(content_disposition,
+                encoding='latin-1', errors='replace').split(';')[1].split('=')[1]
             filename = filename.strip('"\'')
             return self.from_filename(filename)
         except IndexError:


Test patch:
diff --git a/tests/test_responsetypes.py b/tests/test_responsetypes.py
--- a/tests/test_responsetypes.py
+++ b/tests/test_responsetypes.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 import unittest
 from scrapy.responsetypes import responsetypes
 
@@ -20,8 +21,14 @@ def test_from_filename(self):
 
     def test_from_content_disposition(self):
         mappings = [
-            ('attachment; filename="data.xml"', XmlResponse),
-            ('attachment; filename=data.xml', XmlResponse),
+            (b'attachment; filename="data.xml"', XmlResponse),
+            (b'attachment; filename=data.xml', XmlResponse),
+            (u'attachment;filename=data£.tar.gz'.encode('utf-8'), Response),
+            (u'attachment;filename=dataµ.tar.gz'.encode('latin-1'), Response),
+            (u'attachment;filename=data高.doc'.encode('gbk'), Response),
+            (u'attachment;filename=دورهdata.html'.encode('cp720'), HtmlResponse),
+            (u'attachment;filename=日本語版Wikipedia.xml'.encode('iso2022_jp'), XmlResponse),
+
         ]
         for source, cls in mappings:
             retcls = responsetypes.from_content_disposition(source)