Repository: pypa/pip

Issue:
Local HTTP cache is confusing for package maintainers
I've seen this issue raised multiple times in different formats and mediums. Most recently on #pypa today, this question was asked:

> Tried this twice today, is it unusual? New releases to PyPi take 10+ minutes before they can be installed via pip.

The reason for this is that pip uses [CacheControl](https://pypi.org/project/CacheControl/) to maintain a local HTTP cache, and it will not retry requests made within the last 10 minutes:

https://github.com/pypa/pip/blob/e818039dde54eae19391c829e4ffd227a988d76c/src/pip/_internal/index.py#L781-L787

Unfortunately, for a lot of package authors/maintainers, this is a common workflow:

1. The have recently (< 10 minutes ago) run`pip install my_package`
1. They publish a new version of `my_package`
1. They then immediately run `pip install my_package` to try and install the new version
1. `pip` says it can't find the new version (because the cache isn't old enough)

I think that this creates enough confusion about PyPI being "slow" or some type of user error that it's worth addressing in some way.

Some ideas for how this could be addressed:

1. Disable the local cache entirely: this would probably be overkill, I don't think we should do that.
1. Shorten the `max-age` for cache entries: This just shortens the window for when this could happen, but doesn't fix the underlying issue.
1. Leave the cache, but add some extra API to determine if a project has been updated in the last 10 minutes: this would be a lot of work for both `pip` and PyPI maintainers, plus the response for this endpoint would be essentially un-cacheable (from both `pip` and PyPI's perspective)
1. Tell users to use `--no-cache`: This works, but there isn't a great way to inform new users "this is what's happening" and "this is how to fix it" aside from doing it ad-hoc -- it's essentially where we are right now.
1. Exclude the project pages from cache: project pages are pretty light, and always fetching them shouldn't cause an excessive increase in bandwidth or response time. I'm not sure if this is easy to do within `pip`, but seems like the best approach to me.


Discussion:
Agreed that caching is great and very useful! Also for maintainers and developers! For instance when running Tox, we don't want to eagerly hit the online endpoints for each and every dependency.

**Suggestion to improve cache invalidation**: If using editable mode (`pip -e`) and a version of a requirement specification isn't found in the local cache, then switch off the cache and try again (or add an error message hint to use `--no-cache`).
Slightly puckish suggestion: `pip upload` should clear the cache.
(5) is pretty easy to do (in fact we already do this! We just have it limited to 10 minutes instead of setting ``no-cache`` or whatever).

The main reason it caches at all is just to avoid the case of where ``tox`` might hit the same page 4-10 times in a row for each environment, when generally the page never changes. This is more useful in places with high latency internet connections where installing a large number of items could easily add a non trivial amount of time to the install process. (e.g. if you have 100 things, and it takes 500ms to round trip to PyPI, you're looking at an extra ~minute per tox environment.

I originally chose 10 minutes because I thought it represented a good trade off in terms of how quickly a new package is available and how rarely people release new packages. I'm not married to do the idea though, just explaining the thought process of why it is like it is.

If we do change it, then I suggest changing it in such a way that CacheControl is still caching the page, and doing a conditional GET with the ``ETag`` header. While that won't solve the latency issue, it is still friendlier to people with metered internet to limit the amount of bandwidth we're consuming. 
@dstufft I think everybody is on the same page on the benefits of caching! :) I really <3 that `pip` is sensitive to all sorts of internet connections.

@njsmith 

> Slightly puckish suggestion: pip upload should clear the cache.

This isn't enough, since us confused maintainers will release project A and then wonder why it isn't found in CIs and in other project B that depend on A.

For my use, it would be enough if `pip` simply said _oh I couldn't find a specified version (pinning or version range), let me try without the local cache_. So not entirely switching off caching or changing the timeouts - but automatically switching off caching when it seems evident that a newer version was requested and not found in the local cache.
Another instance of this confusing a user: https://github.com/pypa/pypi-legacy/issues/739#issuecomment-411189941

Resolving patch:
diff --git a/src/pip/_internal/index.py b/src/pip/_internal/index.py
--- a/src/pip/_internal/index.py
+++ b/src/pip/_internal/index.py
@@ -783,7 +783,20 @@ def get_page(cls, link, skip_archives=True, session=None):
                 url,
                 headers={
                     "Accept": "text/html",
-                    "Cache-Control": "max-age=600",
+                    # We don't want to blindly returned cached data for
+                    # /simple/, because authors generally expecting that
+                    # twine upload && pip install will function, but if
+                    # they've done a pip install in the last ~10 minutes
+                    # it won't. Thus by setting this to zero we will not
+                    # blindly use any cached data, however the benefit of
+                    # using max-age=0 instead of no-cache, is that we will
+                    # still support conditional requests, so we will still
+                    # minimize traffic sent in cases where the page hasn't
+                    # changed at all, we will just always incur the round
+                    # trip for the conditional GET now instead of only
+                    # once per 10 minutes.
+                    # For more information, please see pypa/pip#5670.
+                    "Cache-Control": "max-age=0",
                 },
             )
             resp.raise_for_status()


Test patch: