Repository: pypa/pip
Pull request #9498 (2021-01-23 21:56:01)

session: Cache http requests of trusted hosts
As a follow-up to #7885, I want to also cache http responses if the host is specified as trusted. Back in the day, I was not sure how to do it, and I am still unsure whether my solution here works or not. I am welcoming a discussion about this change :smile: 

uranusjr: The idea does not sound unreasonable to me, but I’m a bit worried about security. The cache is global (to the user at least), while `--trusted-host` is per-command, so ideally we should only be able to hit the cache when `--trusted-host` is again given on a subsequent command. I’m not sure if that’s even possible though.
NoahGorny: > 
> 
> The idea does not sound unreasonable to me, but I’m a bit worried about security. The cache is global (to the user at least), while `--trusted-host` is per-command, so ideally we should only be able to hit the cache when `--trusted-host` is again given on a subsequent command. I’m not sure if that’s even possible though.

If you cache a package from a host you specified as trusted- it should be trusted just any other package. When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users, which do not get any caching whatsoever now.

@uranusjr I am also unsure whether this solution works at all, and I am not quite sure how can I test it

uranusjr: > When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users

The problem is exactly that misuse may hurt valid users. If something (say a script) calls pip under the hood, it can pollute the cache and affect all subsequent pip calls without the user knowing.
NoahGorny: > > When people abuse this flag and misuse it- it's a different problem, but it should not hurt the valid users
> 
> The problem is exactly that misuse may hurt valid users. If something (say a script) calls pip under the hood, it can pollute the cache and affect all subsequent pip calls without the user knowing.

I do not think that scripts should ever use `trusted-host` unless created by the same user that is using the script. I think that we can maybe add a warning about the dangers of `trusted-host` when invoking `pip`, and this way we can notify the user in case a script is secretly using it
NoahGorny: Hi @uranusjr, I have updated relevant tests, and I will be glad if you can take another look on this PR :smile: 
pradyunsg: Do we cache any other `http` protocol responses already?
uranusjr: We already cache HTTPS with a valid certificate. This adds caching to domains explicitly trusted by the user (via `--trust-host`).
pradyunsg: ACK. I have no blocking concerns based on a cursory look.

I think I'm ambivalent on this and, thus, I'll defer to someone else to click the merge button. :)

NoahGorny: I had troubles testing this..
I tried to set up a local pypi, and disabled the default "secure" status for localhost.
However, I saw that if we build the package from source, we cache the resulting wheel regardless of my change.
If we do not build from source, and use a wheel, the package is not cached with/without this PR, even when the adapter is the correct `InsecureCacheControlAdapter`. The package is just not cached..

Steps to reproduce:
```bash
# from source
pip install example
pip cache list | grep example
pip cache remove example; pip uninstall example

# set up local pypi with the tar, remove it from SECURE_ORIGINS
pip install example --index-url http://localhost:8080 --trusted-host localhost example
# still works, regardless of this PR or not
pip cache list | grep example
pip cache remove example; pip uninstall example
# Save the created wheel file and put in the pypi's packages dir
pip install example --index-url http://localhost:8080 --trusted-host localhost example
# Nothing is cached
pip cache list | grep example
```

Any help on this would be appreciated @uranusjr @pradyunsg 
uranusjr: I think you’re checking the wrong cache. pip has an HTTP cache that caches all incoming HTTP responses in `$cache/http`, and a wheel cache that stores built wheels in `$cache/wheels`. You’re adding into the former, but `pip cache` is querying the latter.
NoahGorny: > I think you’re checking the wrong cache. pip has an HTTP cache that caches all incoming HTTP responses in `$cache/http`, and a wheel cache that stores built wheels in `$cache/wheels`. You’re adding into the former, but `pip cache` is querying the latter.

Oh, okay
Is there any way to interact with the http cache?
uranusjr: The information is not exposed to end users, but for tests you can use `SafeFileCache` from `pip._internal.network.cache`, or implement your own with `CacheControl`.
NoahGorny: Howdy!
Sorry for the big delay @uranusjr @pradyunsg
I tested this and it seems like the wanted solution!

Let me know what you think :smile:

Patch:
diff --git a/news/9498.feature.rst b/news/9498.feature.rst
new file mode 100644
index 000000000..0682915e4
--- /dev/null
+++ b/news/9498.feature.rst
@@ -0,0 +1,1 @@
+If a host is explicitly specified as trusted by the user (via the --trusted-host option), cache HTTP responses from it in addition to HTTPS ones.
diff --git a/src/pip/_internal/network/session.py b/src/pip/_internal/network/session.py
index fe1cf717e..e2916ca81 100644
--- a/src/pip/_internal/network/session.py
+++ b/src/pip/_internal/network/session.py
@@ -358,8 +358,15 @@ class PipSession(requests.Session):
         if host_port not in self.pip_trusted_origins:
             self.pip_trusted_origins.append(host_port)
 
+        self.mount(
+            build_url_from_netloc(host, scheme="http") + "/", self._trusted_host_adapter
+        )
         self.mount(build_url_from_netloc(host) + "/", self._trusted_host_adapter)
         if not host_port[1]:
+            self.mount(
+                build_url_from_netloc(host, scheme="http") + ":",
+                self._trusted_host_adapter,
+            )
             # Mount wildcard ports for the same host.
             self.mount(build_url_from_netloc(host) + ":", self._trusted_host_adapter)
 


Test patch:
diff --git a/tests/unit/test_network_session.py b/tests/unit/test_network_session.py
index dd99bc758..1f333fedf 100644
--- a/tests/unit/test_network_session.py
+++ b/tests/unit/test_network_session.py
@@ -82,6 +82,7 @@ class TestPipSession:
         # Check that the "port wildcard" is present.
         assert "https://example.com:" in session.adapters
         # Check that the cache is enabled.
+        assert hasattr(session.adapters["http://example.com/"], "cache")
         assert hasattr(session.adapters["https://example.com/"], "cache")
 
     def test_add_trusted_host(self):
@@ -93,12 +94,20 @@ class TestPipSession:
         prefix3 = "https://host3/"
         prefix3_wildcard = "https://host3:"
 
+        prefix2_http = "http://host2/"
+        prefix3_http = "http://host3/"
+        prefix3_wildcard_http = "http://host3:"
+
         # Confirm some initial conditions as a baseline.
         assert session.pip_trusted_origins == [("host1", None), ("host3", None)]
         assert session.adapters[prefix3] is trusted_host_adapter
         assert session.adapters[prefix3_wildcard] is trusted_host_adapter
 
+        assert session.adapters[prefix3_http] is trusted_host_adapter
+        assert session.adapters[prefix3_wildcard_http] is trusted_host_adapter
+
         assert prefix2 not in session.adapters
+        assert prefix2_http not in session.adapters
 
         # Test adding a new host.
         session.add_trusted_host("host2")
@@ -110,6 +119,7 @@ class TestPipSession:
         # Check that prefix3 is still present.
         assert session.adapters[prefix3] is trusted_host_adapter
         assert session.adapters[prefix2] is trusted_host_adapter
+        assert session.adapters[prefix2_http] is trusted_host_adapter
 
         # Test that adding the same host doesn't create a duplicate.
         session.add_trusted_host("host3")
@@ -121,6 +131,7 @@ class TestPipSession:
 
         session.add_trusted_host("host4:8080")
         prefix4 = "https://host4:8080/"
+        prefix4_http = "http://host4:8080/"
         assert session.pip_trusted_origins == [
             ("host1", None),
             ("host3", None),
@@ -128,6 +139,7 @@ class TestPipSession:
             ("host4", 8080),
         ]
         assert session.adapters[prefix4] is trusted_host_adapter
+        assert session.adapters[prefix4_http] is trusted_host_adapter
 
     def test_add_trusted_host__logging(self, caplog):
         """