pypa/pip — pull request #483 by ptone (2012-03-14 20:04:59+00:00)
Title: Threaded page getting

Currently the function at pip.index.PackageFinder._get_pages is intended to be a threaded operation, but in many to most cases is not.

The issue is that the initial pool of threads is set only based on the number of initial locations to check.

So if the req is ie "markdown"

a single page is initially used (http://pypi.python.org/simple/Markdown/)

This results in a thread pool of 1

When that one page is analyzed for links, they are added to the work queue and that one thread then will need to get them all, one at a time. The attached pull basically looks at the current queue length each time a page is added, and if warranted adds another thread to get pages up to the max of 10.

Since the python-requests project async pool has a dependency on gevent - it is not particularly suited to solve this problem for pip once pip is using requests.

The work done is exactly the same, the only thing this changes is the threaded nature of the process. As such - testing this is not going to be easy. The whole PackageFinder is also pretty poorly tested, so I'm looking for advice on how to handle the testing for this.

This uses task_done() and queue.join() so requires at least 2.5 - don't know when pip will be dropping 2.4, but I'm guessing it is soon.

based on some really rough comparisons - this can yield speedups of 30-50 %

This is related to #371 and would close that issue.