umpirsky/centipede-crawler — pull request #3 by umpirsky (2014-12-20 08:54:01+00:00)
Title: Use asynchronous crawling

Currently, the crawler is running synchronously, meaning that it spends most of its time waiting for IO.
Using an asynchronous HTTP client would allow being much more efficient, by doing HTTP calls in parallel.

Note that the asynchronous behavior can be implemented in a fully BC way: ``$crawler->crawl()`` could still block until all URLs have been crawled, and the callback-based API can be kept the same.

This means that instead of using Goutte to send the calls, you would need to use Guzzle directly, to be able to access its [asynchronous API](http://docs.guzzlephp.org/en/latest/faq.html#can-guzzle-send-asynchronous-requests). You could then convert the response back to a BrowserKit response before calling the callback if you want to keep BC (or you could decide to break BC and to pass a Guzzle response instead)