Repository: scrapy/scrapy

Issue:
Scrapy 0.25 hangs closing spider
A basic spider like this hangs and is only possible to close the spider by double Ctrl-C

``` python
class TestSpider1c(scrapy.Spider):
    name = "test1c"
    allowed_domains = ['productlibrary.brandbank.com']

    start_urls = [
        'https://productlibrary.brandbank.com/products/detail/949211',

    ]

    def parse(self, request):
        return []
```

``` bash
$ scrapy crawl test1c
2014-12-16 11:18:18-0200 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2014-12-16 11:18:18-0200 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2014-12-16 11:18:18-0200 [scrapy] INFO: Enabled item pipelines: 
2014-12-16 11:18:18-0200 [test1c] INFO: Spider opened
2014-12-16 11:18:18-0200 [test1c] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2014-12-16 11:18:18-0200 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023
2014-12-16 11:18:18-0200 [scrapy] DEBUG: Web service listening on 127.0.0.1:6080
2014-12-16 11:18:18-0200 [scrapy] INFO: Scrapy 0.25.1 started (bot: hangtest)
2014-12-16 11:18:18-0200 [scrapy] INFO: Scrapy 0.25.1 started (bot: hangtest)
2014-12-16 11:18:18-0200 [scrapy] INFO: Optional features available: ssl, http11
2014-12-16 11:18:18-0200 [scrapy] INFO: Optional features available: ssl, http11
2014-12-16 11:18:18-0200 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'hangtest.spiders', 'SPIDER_MODULES': ['hangtest.spiders'], 'BOT_NAME': 'hangtest'}
2014-12-16 11:18:18-0200 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'hangtest.spiders', 'SPIDER_MODULES': ['hangtest.spiders'], 'BOT_NAME': 'hangtest'}
2014-12-16 11:18:19-0200 [test1c] DEBUG: Redirecting (302) to <GET https://secure.brandbank.com/users/issue.aspx?wa=wsignin1.0&wtrealm=https%3a%2f%2fproductlibrary.brandbank.com&wctx=https%3a%2f%2fproductlibrary.brandbank.com%2fproducts%2fdetail%2f949211> from <GET https://productlibrary.brandbank.com/products/detail/949211>
2014-12-16 11:18:21-0200 [test1c] DEBUG: Redirecting (302) to <GET https://secure.brandbank.com/login.aspx?ReturnUrl=%2fusers%2fissue.aspx%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252fproductlibrary.brandbank.com%26wctx%3dhttps%253a%252f%252fproductlibrary.brandbank.com%252fproducts%252fdetail%252f949211&wa=wsignin1.0&wtrealm=https%3a%2f%2fproductlibrary.brandbank.com&wctx=https%3a%2f%2fproductlibrary.brandbank.com%2fproducts%2fdetail%2f949211> from <GET https://secure.brandbank.com/users/issue.aspx?wa=wsignin1.0&wtrealm=https%3a%2f%2fproductlibrary.brandbank.com&wctx=https%3a%2f%2fproductlibrary.brandbank.com%2fproducts%2fdetail%2f949211>
2014-12-16 11:18:21-0200 [test1c] DEBUG: Crawled (200) <GET https://secure.brandbank.com/login.aspx?ReturnUrl=%2fusers%2fissue.aspx%3fwa%3dwsignin1.0%26wtrealm%3dhttps%253a%252f%252fproductlibrary.brandbank.com%26wctx%3dhttps%253a%252f%252fproductlibrary.brandbank.com%252fproducts%252fdetail%252f949211&wa=wsignin1.0&wtrealm=https%3a%2f%2fproductlibrary.brandbank.com&wctx=https%3a%2f%2fproductlibrary.brandbank.com%2fproducts%2fdetail%2f949211> (referer: None)
2014-12-16 11:18:21-0200 [test1c] INFO: Closing spider (finished)
2014-12-16 11:18:21-0200 [test1c] INFO: Dumping Scrapy stats:
    {'downloader/request_bytes': 1201,
     'downloader/request_count': 3,
     'downloader/request_method_count/GET': 3,
     'downloader/response_bytes': 6838,
     'downloader/response_count': 3,
     'downloader/response_status_count/200': 1,
     'downloader/response_status_count/302': 2,
     'finish_reason': 'finished',
     'finish_time': datetime.datetime(2014, 12, 16, 13, 18, 21, 833571),
     'log_count/DEBUG': 5,
     'log_count/INFO': 9,
     'response_received_count': 1,
     'scheduler/dequeued': 3,
     'scheduler/dequeued/memory': 3,
     'scheduler/enqueued': 3,
     'scheduler/enqueued/memory': 3,
     'start_time': datetime.datetime(2014, 12, 16, 13, 18, 18, 551032)}
2014-12-16 11:18:21-0200 [test1c] INFO: Spider closed (finished)
^C^C
$
```

git bisect between 0.25.0 and master branch:

> There are only 'skip'ped commits left to test.
> The first bad commit could be any of:
> 39c6a80f9db6ec04cac59f116ee9620c3d540be0
> d7038b2a136a2b79df9fb16d7b3327cb29f9c46f
> 3ae971468ff3a6712aa47ad3a5d2b9f0c9663b60
> 980e30a18758f21f416036c2076bf98630c70193
> a995727117d10133a20553a648e85970fc6a6543
> d40273561dad76a409f847b5f8ce1daafdb1dc7c
> 870438e5f4665de836f7ff423055895b305a4e7f
> eb0253e5301ea54d4f37da8974d0dac295ebe871
> 84fa004793cb1be07c7a3d0ac6fd80a83b4e8487
> d0edad4b0bd93ed34a680ddd6563387be7797128
> 89df18bd2bc6fdc7f2084454a2c69f4db03008ad
> We cannot bisect more!



Discussion:
@dangra #708

@nramirezuy but in this case start_requests output should be an standard iterable based on start_urls. how is #708 supposed to fix the issue?

@dangra You are right, this is different.

[I found this is not being used anywhere.](https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L125)


Resolving patch:
diff --git a/scrapy/core/downloader/handlers/http11.py b/scrapy/core/downloader/handlers/http11.py
--- a/scrapy/core/downloader/handlers/http11.py
+++ b/scrapy/core/downloader/handlers/http11.py
@@ -32,6 +32,7 @@ def __init__(self, settings):
         self._contextFactory = self._contextFactoryClass()
         self._default_maxsize = settings.getint('DOWNLOAD_MAXSIZE')
         self._default_warnsize = settings.getint('DOWNLOAD_WARNSIZE')
+        self._disconnect_timeout = 1
 
     def download_request(self, request, spider):
         """Return a deferred for the HTTP download"""
@@ -41,7 +42,24 @@ def download_request(self, request, spider):
         return agent.download_request(request)
 
     def close(self):
-        return self._pool.closeCachedConnections()
+        d = self._pool.closeCachedConnections()
+        # closeCachedConnections will hang on network or server issues, so
+        # we'll manually timeout the deferred.
+        #
+        # Twisted issue addressing this problem can be found here:
+        # https://twistedmatrix.com/trac/ticket/7738.
+        #
+        # closeCachedConnections doesn't handle external errbacks, so we'll
+        # issue a callback after `_disconnect_timeout` seconds.
+        delayed_call = reactor.callLater(self._disconnect_timeout, d.callback, [])
+
+        def cancel_delayed_call(result):
+            if delayed_call.active():
+                delayed_call.cancel()
+            return result
+
+        d.addBoth(cancel_delayed_call)
+        return d
 
 
 class TunnelError(Exception):


Test patch: