scrapy/scrapy — pull request #109 by redapple (2012-04-07 21:27:11+00:00)
Title: Use twisted.web.client.Agent for download requests (use of HTTP/1.1)

Lately, I've been fiddling with Downloaders, wanting to use HTTP/1.1 and hopefully later benefit from persistent connections once they're officially in Twisted.

So here's my attempt at refactoring HttpDownloadHandler using twisted.web.client.Agent (which uses HTTP/1.1 starting from Twisted 10?) with the Scrapy specificities I could understand in ScrapyHTTPClientFactory.

For now, I added the code to scrapy/core/downloader/handlers/http11.py
but this could be moved to scrapy/contrib/downloaders/ or something.

The test cases I added in test_downloader_handlers.py are OK on my dev machine.

The code still has debug traces.

What you think?