Repository: pex-tool/pex

Issue:
the pip-powered resolve in pex 2 will re-tokenize --find-links pages on each transitive requirement
We've been running into an issue when trying to test the feature in pantsbuild/pants#8793 (inspired by #789) with pex versions later than 2. When the vendored pip resolver fetches html pages provided as `--find-links` arguments, it appears to always fetch with the `cachecontrol` header `max-age: 0`, which appears to mean it always re-fetches and re-tokenizes every `--find-links` html page every time it tries to resolve any requirement. This leads to extremely long resolve times when resolving against a large remote `--find-links` html page in pex 2 (12 minutes vs 1.5 minutes for a particular intransitive resolve).

Application of the `max-age` header: https://github.com/pantsbuild/pex/blob/e078b88659b2992c839c110ca3447f64e6838f08/pex/vendor/_vendored/pip/pip/_internal/index/collector.py#L130-L143

Output excerpt with `-vvvvvvvvv`:
```
Fetching project page and analyzing links: https://INTERNAL-URL.com/INTERNAL-FIND-LINKS-REPO
  Getting page https://INTERNAL-URL.com/INTERNAL-FIND-LINKS-REPO
  Looking up "https://INTERNAL-URL.com/INTERNAL-FIND-LINKS-REPO" in the cache
  Request header has "max_age" as 0, cache bypassed
  Resetting dropped connection: INTERNAL-URL.com
  https://INTERNAL-URL.com:443 "GET /INTERNAL-FIND-LINKS-REPO HTTP/1.1" 200 858
  Updating cache with response from "https://INTERNAL-URL.com/INTERNAL-FIND-LINKS-REPO"
```

Size of our large `--find-links` page (when saved to disk):
```bash
> wc -c wow.html
1577275 wow.html
```

Will try to post a repro if possible which avoids dumping our internal repo into a public issue. Note that pex 1 does not appear to support local fileystem paths to html pages with `--find-links`, so `--find-links=wow.html` in pex 2 is not directly comparable to the result in pex 1.


Discussion:
Ok, have managed to find a nice repro for this by generating a massive `--find-links` page. Executing `chmod +x wow.sh && ./wow.sh` from this gist: https://gist.github.com/cosmicexplorer/7066947bcb37d3f44863796841c76998 should demonstrate a sizable regression when resolving `tensorflow==1.14.0` from pex `1.6.12` to pex `2.1.2`.
Ok, a PR has been created upstream at pypa/pip#7729, and when the changes from our pip fork at pantsbuild/pip#5 are merged in, this issue will be resolved.

Resolving patch:
diff --git a/pex/vendor/__init__.py b/pex/vendor/__init__.py
--- a/pex/vendor/__init__.py
+++ b/pex/vendor/__init__.py
@@ -99,7 +99,7 @@ def iter_vendor_specs():
 
   # We shell out to pip at buildtime to resolve and install dependencies.
   # N.B.: This is pip 20.0.dev0 with a patch to support foreign download targets more fully.
-  yield VendorSpec.vcs('git+https://github.com/pantsbuild/pip@e96681a0d5cf#egg=pip', rewrite=False)
+  yield VendorSpec.vcs('git+https://github.com/pantsbuild/pip@5eb9470c0c59#egg=pip', rewrite=False)
 
   # We expose this to pip at buildtime for legacy builds, but we also use pkg_resources via
   # pex.third_party at runtime in various ways.
diff --git a/pex/vendor/_vendored/pip/pip/_internal/index/collector.py b/pex/vendor/_vendored/pip/pip/_internal/index/collector.py
--- a/pex/vendor/_vendored/pip/pip/_internal/index/collector.py
+++ b/pex/vendor/_vendored/pip/pip/_internal/index/collector.py
@@ -24,8 +24,8 @@
 
 if MYPY_CHECK_RUNNING:
     from typing import (
-        Callable, Iterable, List, MutableMapping, Optional, Sequence, Tuple,
-        Union,
+        Any, Callable, Dict, Iterable, List, MutableMapping, Optional,
+        Sequence, Tuple, Union,
     )
     import xml.etree.ElementTree
 
@@ -41,6 +41,32 @@
 logger = logging.getLogger(__name__)
 
 
+def lru_cache(
+        *args,                  # type: Any
+        **kwargs                # type: Any
+):
+    # type: (...) -> Any
+    cache = {}                  # type: Dict[Any, Any]
+
+    def wrapper(fn):
+        # type: (Any) -> Any
+
+        def wrapped(
+                *args,          # type: Any
+                **kwargs        # type: Any
+        ):
+            # type: (...) -> Any
+            cache_key = tuple(args) + tuple(kwargs.items())
+            value = cache.get(cache_key, None)
+            if value is not None:
+                return value
+            value = fn(*args, **kwargs)
+            cache[cache_key] = value
+            return value
+        return wrapped
+    return wrapper
+
+
 def _match_vcs_scheme(url):
     # type: (str) -> Optional[str]
     """Look for VCS schemes in the URL.
@@ -244,6 +270,38 @@ def _create_link_from_element(
     return link
 
 
+class CacheablePageContent(object):
+    def __init__(self, page):
+        # type: (HTMLPage) -> None
+        self.page = page
+
+    def __eq__(self, other):
+        # type: (object) -> bool
+        return (isinstance(other, type(self)) and
+                self.page.content == other.page.content and
+                self.page.encoding == other.page.encoding)
+
+    def __hash__(self):
+        # type: () -> int
+        return hash((self.page.content, self.page.encoding))
+
+
+def with_cached_html_pages(fn):
+    # type: (Any) -> Any
+
+    @lru_cache(maxsize=None)
+    def wrapper(cacheable_page):
+        # type: (CacheablePageContent) -> List[Any]
+        return list(fn(cacheable_page.page))
+
+    def wrapper_wrapper(page):
+        # type: (HTMLPage) -> List[Any]
+        return wrapper(CacheablePageContent(page))
+
+    return wrapper_wrapper
+
+
+@with_cached_html_pages
 def parse_links(page):
     # type: (HTMLPage) -> Iterable[Link]
     """
@@ -308,6 +366,18 @@ def _make_html_page(response):
     return HTMLPage(response.content, encoding=encoding, url=response.url)
 
 
+def with_cached_link_fetch(fn):
+    # type: (Any) -> Any
+
+    @lru_cache(maxsize=None)
+    def wrapper(link, session=None):
+        # type: (Link, Optional[PipSession]) -> Optional[HTMLPage]
+        return fn(link, session=session)
+
+    return wrapper
+
+
+@with_cached_link_fetch
 def _get_html_page(link, session=None):
     # type: (Link, Optional[PipSession]) -> Optional[HTMLPage]
     if session is None:


Test patch: