Repository: pypa/pip

Issue:
--trusted-host in requirements.txt is ignored
According to the release notes and the old issue #2822, trusted-host should work in a requirements.txt file, but it has no effect. I can't find it documented but since it doesn't generate a warning, I assume it's still supported.
- Pip version: 8.1.2
- Python version: 2.7.5
- Operating System: CentOS 7.2.1511

This is the requirements.txt file:

```
--trusted-host doubly.so
--index-url https://doubly.so/pub/py/simple
dummy_test
```

It does not work unless I give the trusted-host option on the command-line:

``` bash
[14:06:19] pip install -r requirements.txt 
DEPRECATION: --allow-all-external has been deprecated and will be removed in the future. Due to changes in the repository protocol, it no longer has any effect.
Collecting dummy_test (from -r requirements.txt (line 3))
  Could not fetch URL https://doubly.so/pub/py/simple/dummy-test/: There was a problem confirming the ssl certificate: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:765) - skipping
  Could not find a version that satisfies the requirement dummy_test (from -r requirements.txt (line 3)) (from versions: )
No matching distribution found for dummy_test (from -r requirements.txt (line 3))

[1][14:06:21] pip install --trusted-host doubly.so -r requirements.txt 
DEPRECATION: --allow-all-external has been deprecated and will be removed in the future. Due to changes in the repository protocol, it no longer has any effect.
Collecting dummy_test (from -r requirements.txt (line 3))
Installing collected packages: dummy-test
Successfully installed dummy-test-0.1.3
```



Discussion:
I am also having this issue. Based on a quick scan of pip source, I do not believe `--trusted-host` is being ignored per se in requirements.txt, just that it is treated differently than it is on the command line, which is definitely confusing and possibly dangerous.

It appears (again, quick scan) that on the command line `--trusted-host` bypasses cert validation by specially setting up `InsecureHTTPAdapter`, whereas in requirements.txt it is merely added to `finder.secure_origins` which is aimed at using http rather than https URLs. These are two quite different things.

The fact that pip uses two different mechanisms for the same option is disturbing.

**trusted-host** options is for command-line or **pip.conf** only. It has no business being allowed in **requirements.txt** as that defeats the purpose of protecting your deployment/build etc.

If you are worried about security then `index-url` in requirements.txt is the bigger threat. There is no point in spoofing an HTTPS connection when you can just use a custom url.

Malicious `requirements.txt` files are not part of the threat model pip operates under. The only way a `requirements.txt` file gets read is if the end user includes it explicitly via `-r` to the pip command line or via `-r` in a `requirements.txt` that was explicitly included via the pip command line.

This seems to still be an issue for me with Pip 9.0.1.

Are there any thoughts about what the correct behavior should be? For me the behavior that occurs when `--trusted-host` is on the command line (using `InsecureHTTPAdapter`) is better since that allows for self-signed certificates, but I'm worried there might be issues that I'm not seeing.
Out of the two current behaviours, I'm not sure which one is better. I guess that discussion is something we can defer until someone actually has the interest to opens a PR for this. For that reason, I've labelled this issue as an "deferred till PR".

This label is essentially for indicating that further discussion related to this issue should be deferred until someone comes around to make a PR. This does not mean that the said PR would be accepted - it has not been determined the change would be useful to pip and that decision has been deferred until the PR is made.


Resolving patch:
diff --git a/src/pip/_internal/download.py b/src/pip/_internal/download.py
--- a/src/pip/_internal/download.py
+++ b/src/pip/_internal/download.py
@@ -588,6 +588,8 @@ def __init__(self, *args, **kwargs):
         # well as any https:// host that we've marked as ignoring TLS errors
         # for.
         insecure_adapter = InsecureHTTPAdapter(max_retries=retries)
+        # Save this for later use in add_insecure_host().
+        self._insecure_adapter = insecure_adapter
 
         self.mount("https://", secure_adapter)
         self.mount("http://", insecure_adapter)
@@ -598,7 +600,11 @@ def __init__(self, *args, **kwargs):
         # We want to use a non-validating adapter for any requests which are
         # deemed insecure.
         for host in insecure_hosts:
-            self.mount("https://{}/".format(host), insecure_adapter)
+            self.add_insecure_host(host)
+
+    def add_insecure_host(self, host):
+        # type: (str) -> None
+        self.mount('https://{}/'.format(host), self._insecure_adapter)
 
     def request(self, method, url, *args, **kwargs):
         # Allow setting a default timeout on a session
diff --git a/src/pip/_internal/index.py b/src/pip/_internal/index.py
--- a/src/pip/_internal/index.py
+++ b/src/pip/_internal/index.py
@@ -608,8 +608,8 @@ def create(
         # type: (...) -> PackageFinder
         """Create a PackageFinder.
 
-        :param trusted_hosts: Domains that we won't emit warnings for when
-            not using HTTPS.
+        :param trusted_hosts: Domains not to emit warnings for when not using
+            HTTPS.
         :param session: The Session to use to make requests.
         :param format_control: A FormatControl object or None. Used to control
             the selection of source packages / binary packages when consulting
@@ -676,12 +676,23 @@ def set_allow_all_prereleases(self):
         # type: () -> None
         self.candidate_evaluator.allow_all_prereleases = True
 
-    def extend_trusted_hosts(self, hosts):
-        # type: (List[str]) -> None
-        for host in hosts:
-            if host in self.trusted_hosts:
-                continue
-            self.trusted_hosts.append(host)
+    def add_trusted_host(self, host, source=None):
+        # type: (str, Optional[str]) -> None
+        """
+        :param source: An optional source string, for logging where the host
+            string came from.
+        """
+        # It is okay to add a previously added host because PipSession stores
+        # the resulting prefixes in a dict.
+        msg = 'adding trusted host: {!r}'.format(host)
+        if source is not None:
+            msg += ' (from {})'.format(source)
+        logger.info(msg)
+        self.session.add_insecure_host(host)
+        if host in self.trusted_hosts:
+            return
+
+        self.trusted_hosts.append(host)
 
     def iter_secure_origins(self):
         # type: () -> Iterator[SecureOrigin]
diff --git a/src/pip/_internal/req/req_file.py b/src/pip/_internal/req/req_file.py
--- a/src/pip/_internal/req/req_file.py
+++ b/src/pip/_internal/req/req_file.py
@@ -250,8 +250,9 @@ def process_line(
             finder.find_links.append(value)
         if opts.pre:
             finder.set_allow_all_prereleases()
-        if opts.trusted_hosts:
-            finder.extend_trusted_hosts(opts.trusted_hosts)
+        for host in opts.trusted_hosts or []:
+            source = 'line {} of {}'.format(line_number, filename)
+            finder.add_trusted_host(host, source=source)
 
 
 def break_args_options(line):


Test patch:
diff --git a/tests/unit/test_index.py b/tests/unit/test_index.py
--- a/tests/unit/test_index.py
+++ b/tests/unit/test_index.py
@@ -163,16 +163,61 @@ def test_create__target_python(self):
         assert actual_target_python is target_python
         assert actual_target_python.py_version_info == (3, 7, 3)
 
-    def test_extend_trusted_hosts(self):
-        trusted_hosts = ['host1', 'host2']
-        finder = make_test_finder(trusted_hosts=trusted_hosts)
-
-        # Check that extend_trusted_hosts() prevents duplicates.
-        finder.extend_trusted_hosts(['host2', 'host3', 'host2'])
-        assert finder.trusted_hosts == ['host1', 'host2', 'host3'], (
+    def test_add_trusted_host(self):
+        # Leave a gap to test how the ordering is affected.
+        trusted_hosts = ['host1', 'host3']
+        session = PipSession(insecure_hosts=trusted_hosts)
+        finder = make_test_finder(
+            session=session,
+            trusted_hosts=trusted_hosts,
+        )
+        insecure_adapter = session._insecure_adapter
+        prefix2 = 'https://host2/'
+        prefix3 = 'https://host3/'
+
+        # Confirm some initial conditions as a baseline.
+        assert finder.trusted_hosts == ['host1', 'host3']
+        assert session.adapters[prefix3] is insecure_adapter
+        assert prefix2 not in session.adapters
+
+        # Test adding a new host.
+        finder.add_trusted_host('host2')
+        assert finder.trusted_hosts == ['host1', 'host3', 'host2']
+        # Check that prefix3 is still present.
+        assert session.adapters[prefix3] is insecure_adapter
+        assert session.adapters[prefix2] is insecure_adapter
+
+        # Test that adding the same host doesn't create a duplicate.
+        finder.add_trusted_host('host3')
+        assert finder.trusted_hosts == ['host1', 'host3', 'host2'], (
             'actual: {}'.format(finder.trusted_hosts)
         )
 
+    def test_add_trusted_host__logging(self, caplog):
+        """
+        Test logging when add_trusted_host() is called.
+        """
+        trusted_hosts = ['host1']
+        session = PipSession(insecure_hosts=trusted_hosts)
+        finder = make_test_finder(
+            session=session,
+            trusted_hosts=trusted_hosts,
+        )
+        with caplog.at_level(logging.INFO):
+            # Test adding an existing host.
+            finder.add_trusted_host('host1', source='somewhere')
+            finder.add_trusted_host('host2')
+            # Test calling add_trusted_host() on the same host twice.
+            finder.add_trusted_host('host2')
+
+        actual = [(r.levelname, r.message) for r in caplog.records]
+        expected = [
+            ('INFO', "adding trusted host: 'host1' (from somewhere)"),
+            ('INFO', "adding trusted host: 'host2'"),
+            ('INFO', "adding trusted host: 'host2'"),
+        ]
+        assert actual == expected
+
     def test_iter_secure_origins(self):
         trusted_hosts = ['host1', 'host2']
         finder = make_test_finder(trusted_hosts=trusted_hosts)
diff --git a/tests/unit/test_req_file.py b/tests/unit/test_req_file.py
--- a/tests/unit/test_req_file.py
+++ b/tests/unit/test_req_file.py
@@ -1,3 +1,4 @@
+import logging
 import os
 import subprocess
 import textwrap
@@ -314,9 +315,21 @@ def test_set_finder_extra_index_urls(self, finder):
         list(process_line("--extra-index-url=url", "file", 1, finder=finder))
         assert finder.index_urls == ['url']
 
-    def test_set_finder_trusted_host(self, finder):
-        list(process_line("--trusted-host=url", "file", 1, finder=finder))
-        assert finder.trusted_hosts == ['url']
+    def test_set_finder_trusted_host(self, caplog, finder):
+        with caplog.at_level(logging.INFO):
+            list(process_line(
+                "--trusted-host=host", "file.txt", 1, finder=finder,
+            ))
+        assert finder.trusted_hosts == ['host']
+        session = finder.session
+        assert session.adapters['https://host/'] is session._insecure_adapter
+
+        # Test the log message.
+        actual = [(r.levelname, r.message) for r in caplog.records]
+        expected = [
+            ('INFO', "adding trusted host: 'host' (from line 1 of file.txt)"),
+        ]
+        assert actual == expected
 
     def test_noop_always_unzip(self, finder):
         # noop, but confirm it can be set