pypa/pip — Issue #371: Threaded downloads - thread pool can only get smaller
Author: PaulMcMillan
Created: 2011-10-17 05:54:36+00:00
State: open

Pip does use separate threads for different resources to speed up spider time while choosing a resource. However, there's a bug which means that that thread pool can only shrink, and never get bigger. This means that if the pool shrinks to 1 thread because a page loads slowly (maybe a project homepage), then all the pages which are added to the pool from that page can only be retrieved by a single thread.

Additionally, the initial location count is probably not a particularly good indicator for how many threads we may need.

Code is pretty self explanatory:
https://github.com/pypa/pip/blob/develop/pip/index.py#L212

The code should probably use either blocking `get()` and call `task_done()`, or be refactored to use threading Condition objects and `wait()`.