pypa/pip — pull request #677 by blaze33 (2012-09-14 18:29:41+00:00)
Title: Explicitly ignore rel='download' links while looking for html pages.

Yesterday I got tired to have to wait 15 to 20 minutes each time I need to reinstall the virtualenv of my current project (also had a bad connection which made this more noticeable). This looks clearly too long considering I have a requirements.txt explicitly defining each package version with each package archive already present in the download cache folder.

Ideally I hoped to find an easy way to firstly check the cache when specifying an explicit revision number but that would require too much work because the cache is currently only checked after analyzing every link of every requested page.

Nevertheless, after profiling several pip installations I found out that `index.PackageFinder._get_queued_page` added each and every link to the queue even those who where explicitly marked as `rel='download'`. Those links are obviously rejected later in the process by `HTMLPage.get_page` when checking content-type != text/html but that costs us a header request for every link.

I propose not to include the `rel='download'` links in the page download queue.

A simple comparison with `pip install -r requirements.txt` shows the following improvement :
*Current develop branch:
```
ncalls tottime percall cumtime percall filename:lineno(function)
59 0.041 0.001 172.957 2.931 index.py:220(_get_pages)
```
*Patched develop branch:
```
59 0.039 0.001 99.354 1.684 index.py:220(_get_pages)
```