scrapy/scrapy — pull request #999 by Curita (2014-12-31 00:09:52+00:00)
Title: Patch hanging HTTPConnectionPool.closeCachedConnections call

This PR fixes #985

The issue while closing the spider on that domain raised while closing the `HTTP11DownloadHandler` (it can be seen [here](https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handlers/http11.py#L44)). The server doesn't close the connection properly so the client waits for a confirmation that doesn't arrive. I've reported this on the Twisted issue tracker ([#7738](https://twistedmatrix.com/trac/ticket/7738)) since this a problem concerning their `HTTPConnectionPool` while cleaning persistent connections, but I've patched it externally by firing a deferred on a DelayedCalled.

This problem hasn't came up before the changes on the crawling API because the reactor was stopped along the download handlers on the engine_stop signal ([CrawlerProcess@8fece4b](https://github.com/scrapy/scrapy/blob/8fece4b0b8eb8772a07673b4166cdcdb5c017eb8/scrapy/crawler.py#L150) and [DownloadHandlers@8fece4b](https://github.com/scrapy/scrapy/blob/8fece4b0b8eb8772a07673b4166cdcdb5c017eb8/scrapy/core/downloader/handlers/__init__.py#L31)). Instead of that, now the reactor is stopped after the crawl deferreds has been fired ([CrawlerProcess](https://github.com/scrapy/scrapy/blob/master/scrapy/crawler.py#L150)), which happens after each engine has stopped, so the `HTTP11DownloadHandler.close` isn't abruptly terminated.

I chose a `_disconnect_timeout` of one second on a tradeoff between the previous instant termination and giving a little time to the connections to close in an orderly manner. It could be a new Scrapy setting, and that's why I set this variable on the class init, but I think that kind of parametrization is not needed right now.

I struggled on mocking a server that mimics this behavior (Twisted doesn't provide a way to do it as they manage the sockets internally, and I'm still not sure how to do it otherwise), so that's why I'm submitting the PR as it is and later I'll try to add a proper test.