Repository: scrapy/scrapy

Issue:
Request with space in url generated 550 error
When processing a request with a space in the url, a 550 error is generated.  

``` python

import scrapy

class TestSpider(scrapy.Spider):
    name = "test"
    handle_httpstatus_list = [404]

    def start_requests(self):
        #This ones does not work
        link = 'ftp://sdrftp03.dor.state.fl.us/Data Books/1976_2009_Data_Book_PDF/00FLpropdata.pdf'

        #This one works
#        link = 'ftp://sdrftp03.dor.state.fl.us/MaximumMillageData/comp07.pdf'

        yield scrapy.Request(link, meta={'ftp_user':'anonymous','ftp_password': 'anonymous'})

    def parse(self, response):
        print response.body
```



Discussion:
550 ftp error means "Access denied" so it can be anything but I can't reproduce the error, are you sure the file was readable by anonymous user?

the same link now returns 404 (not found)

```
scrapy ~$ scrapy runspider i1232.py
2015-05-23 19:55:58+0000 [scrapy] INFO: Scrapy 1.1.0dev1 started (bot: scrapybot)
2015-05-23 19:55:58+0000 [scrapy] INFO: Optional features available: ssl, http11
2015-05-23 19:55:58+0000 [scrapy] INFO: Overridden settings: {}
2015-05-23 19:55:58+0000 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, LogStats, SpiderState
2015-05-23 19:55:58+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-05-23 19:55:58+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-05-23 19:55:58+0000 [scrapy] INFO: Enabled item pipelines: 
2015-05-23 19:55:58+0000 [scrapy] INFO: Spider opened
2015-05-23 19:55:58+0000 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-05-23 19:55:58+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-05-23 19:56:00+0000 [scrapy] DEBUG: Crawled (404) <GET ftp://sdrftp03.dor.state.fl.us/Data%20Books/1976_2009_Data_Book_PDF/00FLpropdata.pdf> (referer: None)
['550 /Data%20Books/1976_2009_Data_Book_PDF/00FLpropdata.pdf: The system cannot find the path specified. ']
2015-05-23 19:56:00+0000 [scrapy] INFO: Closing spider (finished)
2015-05-23 19:56:00+0000 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 279,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 133,
 'downloader/response_count': 1,
 'downloader/response_status_count/404': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 5, 23, 22, 56, 0, 723492),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2015, 5, 23, 22, 55, 58, 594758)}
2015-05-23 19:56:00+0000 [scrapy] INFO: Spider closed (finished)
```

I tried with curl and it downloaded a PDF file.

``` shell
~$ curl -u anonymous:anonymous 'ftp://sdrftp03.dor.state.fl.us/Data Books/1976_2009_Data_Book_PDF/00FLpropdata.pdf' >a.pdf
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  531k  100  531k    0     0  88427      0  0:00:06  0:00:06 --:--:--  108k
~$ file a.pdf 
a.pdf: PDF document, version 1.3
```

Scrapy ftp session as seen by Wireshark:

![screenshot from 2015-05-23 20-02-49](https://cloud.githubusercontent.com/assets/37369/7786014/bced8f48-0186-11e5-8b18-c54ecf36ff2c.png)

Curl ftp session as seen by Wireshark:

![screenshot from 2015-05-23 20-04-56](https://cloud.githubusercontent.com/assets/37369/7786020/03b64f46-0187-11e5-8fcd-f856f9c7bff8.png)

The space in the url is quoted by `w3lib.url.safe_url_string()` function called by Request init method.  

Got it working with the following change. 

``` diff
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
index d96e37f..93fa0a3 100644
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -30,7 +30,7 @@ In case of status 200 request, response.headers will come with two keys:

 import re
 from io import BytesIO
-from six.moves.urllib.parse import urlparse
+from six.moves.urllib.parse import urlparse, unquote

 from twisted.internet import reactor
 from twisted.protocols.ftp import FTPClient, CommandFailed
@@ -73,7 +73,7 @@ class FTPDownloadHandler(object):
                                     request.meta["ftp_password"],
                                     passive=request.meta.get("ftp_passive", 1))
         return creator.connectTCP(parsed_url.hostname, parsed_url.port or 21).addCallback(self.got
-                                request, parsed_url.path)
+                                request, unquote(parsed_url.path))

     def gotClient(self, client, request, filepath):
         self.client = client
```

Scrapy logs:

```
scrapy ~$ scrapy runspider i1232.py
2015-05-23 20:20:19+0000 [scrapy] INFO: Scrapy 1.1.0dev1 started (bot: scrapybot)
2015-05-23 20:20:19+0000 [scrapy] INFO: Optional features available: ssl, http11
2015-05-23 20:20:19+0000 [scrapy] INFO: Overridden settings: {}
2015-05-23 20:20:19+0000 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, CoreStats, LogStats, SpiderState
2015-05-23 20:20:19+0000 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-05-23 20:20:19+0000 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-05-23 20:20:19+0000 [scrapy] INFO: Enabled item pipelines: 
2015-05-23 20:20:19+0000 [scrapy] INFO: Spider opened
2015-05-23 20:20:19+0000 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2015-05-23 20:20:19+0000 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2015-05-23 20:20:24+0000 [scrapy] DEBUG: Crawled (200) <GET ftp://sdrftp03.dor.state.fl.us/Data%20Books/1976_2009_Data_Book_PDF/00FLpropdata.pdf> (referer: None)
%���-1.3...
.....
959 
2015-05-23 20:20:24+0000 [scrapy] INFO: Closing spider (finished)
2015-05-23 20:20:24+0000 [scrapy] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 279,
 'downloader/request_count': 1,
 'downloader/request_method_count/GET': 1,
 'downloader/response_bytes': 544043,
 'downloader/response_count': 1,
 'downloader/response_status_count/200': 1,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2015, 5, 23, 23, 20, 24, 630781),
 'log_count/DEBUG': 2,
 'log_count/INFO': 7,
 'response_received_count': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2015, 5, 23, 23, 20, 19, 750423)}
2015-05-23 20:20:24+0000 [scrapy] INFO: Spider closed (finished)
```

And wireshark session:

![screenshot from 2015-05-23 20-17-23](https://cloud.githubusercontent.com/assets/37369/7786045/c52e9448-0188-11e5-8fb5-10203eceb0ce.png)


Resolving patch:
diff --git a/scrapy/core/downloader/handlers/ftp.py b/scrapy/core/downloader/handlers/ftp.py
--- a/scrapy/core/downloader/handlers/ftp.py
+++ b/scrapy/core/downloader/handlers/ftp.py
@@ -30,7 +30,7 @@
 
 import re
 from io import BytesIO
-from six.moves.urllib.parse import urlparse
+from six.moves.urllib.parse import urlparse, unquote
 
 from twisted.internet import reactor
 from twisted.protocols.ftp import FTPClient, CommandFailed
@@ -73,7 +73,7 @@ def download_request(self, request, spider):
                                     request.meta["ftp_password"],
                                     passive=request.meta.get("ftp_passive", 1))
         return creator.connectTCP(parsed_url.hostname, parsed_url.port or 21).addCallback(self.gotClient,
-                                request, parsed_url.path)
+                                request, unquote(parsed_url.path))
 
     def gotClient(self, client, request, filepath):
         self.client = client


Test patch:
diff --git a/tests/test_downloader_handlers.py b/tests/test_downloader_handlers.py
--- a/tests/test_downloader_handlers.py
+++ b/tests/test_downloader_handlers.py
@@ -511,7 +511,9 @@ def setUp(self):
         os.mkdir(self.directory)
         userdir = os.path.join(self.directory, self.username)
         os.mkdir(userdir)
-        FilePath(userdir).child('file.txt').setContent("I have the power!")
+        fp = FilePath(userdir)
+        fp.child('file.txt').setContent("I have the power!")
+        fp.child('file with spaces.txt').setContent("Moooooooooo power!")
 
         # setup server
         realm = FTPRealm(anonymousRoot=self.directory, userHome=self.directory)
@@ -547,6 +549,19 @@ def _test(r):
             self.assertEqual(r.headers, {'Local Filename': [''], 'Size': ['17']})
         return self._add_test_callbacks(d, _test)
 
+    def test_ftp_download_path_with_spaces(self):
+        request = Request(
+            url="ftp://127.0.0.1:%s/file with spaces.txt" % self.portNum,
+            meta={"ftp_user": self.username, "ftp_password": self.password}
+        )
+        d = self.download_handler.download_request(request, None)
+
+        def _test(r):
+            self.assertEqual(r.status, 200)
+            self.assertEqual(r.body, 'Moooooooooo power!')
+            self.assertEqual(r.headers, {'Local Filename': [''], 'Size': ['18']})
+        return self._add_test_callbacks(d, _test)
+
     def test_ftp_download_notexist(self):
         request = Request(url="ftp://127.0.0.1:%s/notexist.txt" % self.portNum,
                 meta={"ftp_user": self.username, "ftp_password": self.password})