Repository: pypa/pip

Issue:
pip 19.3 doesn't send client certificate
**Ubuntu 18.04 virtual environment**

* pip version: 19.3
* Python version: 3.6.8
* OS: Ubuntu 18.04.3 LTS

We have a private Pypi server hosted with [pypicloud](https://pypicloud.readthedocs.io/en/latest/index.html). We use client certificates to authenticate users for downloading/uploading packages.

**Description**
pip 19.3 doesn't seem to send our client certificates so authentication fails and packages cannot be installed:

`WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)'),)': /simple/<our package name>/
`
I captured some of the SSL traffic from pip install in Wireshark and the client certificate option is there in the  SSL handshake, but the certificates length is 0 with pip 19.3:

![image](https://user-images.githubusercontent.com/9781018/66789548-28f54080-eeba-11e9-8124-315e814564bc.png)

In 19.2.1, the length is non-zero and Wireshark shows the client certificate I expect.

**Expected behavior**
We should not get an SSL error if our client certificates and CA certificates are not expired. I have checked our server logs there don't appear to be any errors there with our certificates.
If I downgrade to pip 19.2.1 or 19.2.3 in my virtual environment, then the SSL error goes away.

I also checked with the `openssl s_client` that a handshake succeeded with the same client certificate:
```
openssl s_client -connect <my server> -cert <cert> -key <key> -state
CONNECTED(00000005)
SSL_connect:before SSL initialization
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS write client hello
SSL_connect:SSLv3/TLS read server hello
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = <my server>
verify return:1
SSL_connect:SSLv3/TLS read server certificate
SSL_connect:SSLv3/TLS read server key exchange
SSL_connect:SSLv3/TLS read server certificate request
SSL_connect:SSLv3/TLS read server done
SSL_connect:SSLv3/TLS write client certificate
...
SSL handshake has read 4268 bytes and written 1546 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 
```

**How to Reproduce**

1. Setup pip.conf or command-line arguments to use client certificate
2. pip install <package>
3. sslv3 alert handshake failure occurs

**Output**

```
pip install <my package>
Looking in indexes: https://pypi.org/simple/, https://<my server>/simple/
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)'),)': /simple/<my package>/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:852)'),)': /simple/<my package>/
```



Discussion:
I cannot reproduce this (Ubuntu 18.04.2, Python 3.6.7) with

<details>
<summary><strong>repro.sh</strong></summary>

```
#!/bin/sh

trap "exit" INT TERM
trap "kill 0" EXIT
set -e

cd "$(mktemp -d)"

openssl req -new -x509 -nodes \
    -out cert.pem -keyout cert.pem \
    -addext 'subjectAltName = IP:127.0.0.1' \
    -subj '/CN=127.0.0.1'

cat <<EOF > server.py
import socket
import ssl
import sys
from pathlib import Path


cert = sys.argv[1]
context = ssl.SSLContext(ssl.PROTOCOL_TLS_SERVER)
context.load_cert_chain(cert, cert)
context.load_verify_locations(cafile=cert)
context.verify_mode = ssl.CERT_REQUIRED


with socket.socket(socket.AF_INET, socket.SOCK_STREAM, 0) as sock:
    sock.bind(('127.0.0.1', 0))
    sock.listen(1)
    _, port = sock.getsockname()
    Path('port.txt').write_text(str(port), encoding='utf-8')
    with context.wrap_socket(sock, server_side=True) as ssock:
        while True:
            conn, addr = ssock.accept()
            cert = conn.getpeercert()
            print(cert)
            conn.write(b'HTTP/1.1 400 Bad Request\r\n\r\n')
            conn.close()
EOF

PYTHON="${PYTHON:-python}"

"$PYTHON" -V

"$PYTHON" -m venv venv

venv/bin/python server.py cert.pem &

sleep 1

venv/bin/python -m pip install --upgrade pip==19.2.3

echo "- Old pip ------------------------------"

venv/bin/python -m pip -V

venv/bin/python -m pip install \
    --ignore-installed \
    --disable-pip-version-check \
    --index-url https://127.0.0.1:$(cat port.txt) \
    --cert cert.pem \
    --client-cert cert.pem \
    pip || true

venv/bin/python -m pip install --upgrade pip

echo "- New pip ------------------------------"

venv/bin/python -m pip -V

pip install \
    --ignore-installed \
    --disable-pip-version-check \
    --index-url https://127.0.0.1:$(cat port.txt) \
    --cert cert.pem \
    --client-cert cert.pem \
    pip
```

</details>

My output is

<details>
<summary><strong>Output</strong></summary>

```
$ PYTHON=~/.pyenv/versions/3.6.7/bin/python ./repro.sh
Generating a RSA private key
................................................................+++++
.......+++++
writing new private key to 'cert.pem'
-----
Python 3.6.7
Collecting pip==19.2.3
  Using cached https://files.pythonhosted.org/packages/30/db/9e38760b32e3e7f40cce46dd5fb107b8c73840df38f0046d8e6514e675a1/pip-19.2.3-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 10.0.1
    Uninstalling pip-10.0.1:
      Successfully uninstalled pip-10.0.1
Successfully installed pip-19.2.3
You are using pip version 19.2.3, however version 19.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
- Old pip ------------------------------
pip 19.2.3 from /tmp/user/1000/tmp.ZqHiG62cpt/venv/lib/python3.6/site-packages/pip (python 3.6)
Looking in indexes: https://127.0.0.1:55649
Collecting pip
{'subject': ((('commonName', '127.0.0.1'),),), 'issuer': ((('commonName', '127.0.0.1'),),), 'version': 3, 'serialNumber': '5D7B2701E9D3E0E8A9E6CA66AEC3849D3BE826CD', 'notBefore': 'Oct 15 01:55:59 2019 GMT', 'notAfter': 'Nov 14 01:55:59 2019 GMT', 'subjectAltName': (('IP Address', '127.0.0.1'),)}
  ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
Collecting pip
  Using cached https://files.pythonhosted.org/packages/4a/08/6ca123073af4ebc4c5488a5bc8a010ac57aa39ce4d3c8a931ad504de4185/pip-19.3-py2.py3-none-any.whl
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3
- New pip ------------------------------
pip 19.3 from /tmp/user/1000/tmp.ZqHiG62cpt/venv/lib/python3.6/site-packages/pip (python 3.6)
Looking in indexes: https://127.0.0.1:55649
Collecting pip
{'subject': ((('commonName', '127.0.0.1'),),), 'issuer': ((('commonName', '127.0.0.1'),),), 'version': 3, 'serialNumber': '5D7B2701E9D3E0E8A9E6CA66AEC3849D3BE826CD', 'notBefore': 'Oct 15 01:55:59 2019 GMT', 'notAfter': 'Nov 14 01:55:59 2019 GMT', 'subjectAltName': (('IP Address', '127.0.0.1'),)}
  ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
```

</details>

Notice in the second instance (with pip 19.3) that the server is still tracing the peer (pip) certificate.

How are you configuring the client cert for pip? Command line, configuration file, or environment variable?

Can you try shaping `repro.sh` from above into something self-contained that demonstrates your issue?
We're using ~/.pip/pip.conf to specify the client certificates. I modified your `repo.sh` and was not able to reproduce the problem using our client + server certificates and a fake SSL server (instead of the python one, I wanted to disable TLS 1.3 so I could see the certificates being sent in Wireshark):

`openssl s_server -accept 8999 -www -cert server.pem -key server.key -CAfile ca-cert.pem -no_tls1_3 -Verify 1`

It's a bit hard to produce something self-contained since we've got a Letsencrypt certificate tied to our own domain and a private PKI infrastructure for the client certificates.

It's looking like it might be an issue when the client certificate bundle is specified in pip.conf, specifying on the command-line seemed to work fine in 19.3. I'll try and come up with a new repro script that simulates this.



You may also run in a container so as not to clobber any existing configuration.
Ok, I think I have a container + script that reproduces the issue. It sets up its own CA and server/client certificates so it should be self-contained. I ran tshark in the Docker container and verified that when pip 19.3 talks to a dummy openssl server acting as pypi.org on the loopback interface, it doesn't send the client cert.

It has something to do with the `trusted-host` parameter in /root/.pip/pip.conf. With that commented out, there's no error. In the output below, some of the output from the openssl s_server process is mixed in with the script output (showing no client certificate sent).

<details>
  <summary>Dockerfile</summary>

```
FROM python:3.8.0-slim-buster
COPY repro.sh /root
COPY pip.conf /root/.pip/pip.conf
WORKDIR /root
```
</details>

<details>
  <summary>pip.conf</summary>

```
[global]
index-url = https://127.0.0.1:8999
trusted-host = 127.0.0.1
client-cert = /root/pip.client.bundle.pem
```
</details>


<details>
  <summary>repro.sh</summary>

```bash
#!/bin/sh

trap "exit" INT TERM
trap "kill 0" EXIT
set -e

# CA + server cert
openssl genrsa -des3 -out ca.key -passout pass:notsecure 2048
openssl req -x509 -new -nodes -key ca.key -sha256 -days 1825 -addext "keyUsage = cRLSign, digitalSignature, keyCertSign" -out ca.pem -subj "/CN=Fake Root CA" -passin pass:notsecure
openssl genrsa -out pip.local.key 2048
openssl req -new -key pip.local.key -out pip.local.csr -subj "/CN=127.0.0.1"

cat << EOF > pip.local.ext
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names

[alt_names]
IP.1 = 127.0.0.1
EOF

openssl x509 -req -in pip.local.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
-out pip.local.pem -days 1825 -sha256 -extfile pip.local.ext -passin pass:notsecure

cat << EOF > pip.client.ext
keyUsage = digitalSignature
extendedKeyUsage = clientAuth
EOF

# client cert
openssl genrsa -out pip.client.key 2048
openssl req -new -key pip.client.key -out pip.client.csr -subj "/CN=pip install"
openssl x509 -req -in pip.client.csr -CA ca.pem -CAkey ca.key -CAcreateserial \
-out pip.client.pem -days 1825 -sha256 -extfile pip.client.ext -passin pass:notsecure

# create key + cert bundle for pip install
cat pip.client.key pip.client.pem > pip.client.bundle.pem

PYTHON="${PYTHON:-python3}"

"$PYTHON" -V

"$PYTHON" -m venv venv

openssl s_server -accept 8999 -www -cert pip.local.pem -key pip.local.key -CAfile ca.pem -no_tls1_3 -Verify 1 &

sleep 1

venv/bin/python -m pip install --index-url https://pypi.org/simple/ --upgrade pip==19.2.3

echo "- Old pip ------------------------------"

venv/bin/python -m pip -V

venv/bin/python -m pip install \
    --ignore-installed \
    --disable-pip-version-check \
    --cert /root/ca.pem \
    pip || true

echo "Upgrading pip --------------------------"

venv/bin/python -m pip install --index-url https://pypi.org/simple/ --upgrade pip

echo "- New pip ------------------------------"

venv/bin/python -m pip -V

pip install \
    --ignore-installed \
    --disable-pip-version-check \
    --cert ca.pem \
    pip
```
</details>

<details>
  <summary>Usage</summary>

```bash
docker build -t pip-debug -f Dockerfile .
docker run -it pip-debug bash
root@6d0a40c1179c:~# ./repro.sh 
```
</details>

<details>
  <summary>Output</summary>

```
root@0e1127dd4124:~# ./repro.sh 
Generating RSA private key, 2048 bit long modulus (2 primes)
.......................+++++
..........+++++
e is 65537 (0x010001)
Generating RSA private key, 2048 bit long modulus (2 primes)
...................................+++++
......................................................................................................................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = 127.0.0.1
Getting CA Private Key
Generating RSA private key, 2048 bit long modulus (2 primes)
........................................+++++
.......................+++++
e is 65537 (0x010001)
Signature ok
subject=CN = pip install
Getting CA Private Key
Python 3.8.0
verify depth is 1, must return a certificate
Using default temp DH parameters
ACCEPT
Looking in indexes: https://pypi.org/simple/
Requirement already up-to-date: pip==19.2.3 in ./venv/lib/python3.8/site-packages (19.2.3)
WARNING: You are using pip version 19.2.3, however version 19.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
- Old pip ------------------------------
pip 19.2.3 from /root/venv/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: https://127.0.0.1:8999
Collecting pip
depth=1 CN = Fake Root CA
verify return:1
depth=0 CN = pip install
verify return:1
  ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
Upgrading pip --------------------------
Looking in indexes: https://pypi.org/simple/
Collecting pip
  Downloading https://files.pythonhosted.org/packages/4a/08/6ca123073af4ebc4c5488a5bc8a010ac57aa39ce4d3c8a931ad504de4185/pip-19.3-py2.py3-none-any.whl (1.4MB)
     |████████████████████████████████| 1.4MB 3.7MB/s 
Installing collected packages: pip
  Found existing installation: pip 19.2.3
    Uninstalling pip-19.2.3:
      Successfully uninstalled pip-19.2.3
Successfully installed pip-19.3
- New pip ------------------------------
pip 19.3 from /root/venv/lib/python3.8/site-packages/pip (python 3.8)
Looking in indexes: https://127.0.0.1:8999
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))': /pip/
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))': /pip/
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))': /pip/
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))': /pip/
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))': /pip/
140716939547776:error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate:../ssl/statem/statem_srvr.c:3672:
Could not fetch URL https://127.0.0.1:8999/pip/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='127.0.0.1', port=8999): Max retries exceeded with url: /pip/ (Caused by SSLError(SSLError(1, '[SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:1108)'))) - skipping
ERROR: Could not find a version that satisfies the requirement pip (from versions: none)
ERROR: No matching distribution found for pip
```
</details>

Nice, thanks.

I bisected and it looks like the issue was introduced in 3f9136f. Previously the "trusted host" parameter with https URLs was only being applied for index URLs that did not have a port specified. As of 19.3 we assume that an unspecified port means the port is a wildcard. That change in conjunction with your configuration may have uncovered a bug in our `InsecureHTTPAdapter` [here](https://github.com/pypa/pip/blob/8c50c8a9bc8579886fa787a631dc15d4b503a8ac/src/pip/_internal/network/session.py#L214-L216) - we aren't doing anything with the `cert` parameter.

If I'm not missing something, I think we should be doing something like

```python
super(InsecureHTTPAdapter, self).cert_verify(conn=conn, url=url, verify=False, cert=cert)
```

to get the correct behavior (from [here](https://github.com/psf/requests/blob/67a7b2e8336951d527e223429672354989384197/requests/adapters.py#L241-L253)).

In your particular case is it possible to drop the trusted-host parameter since it wasn't being applied in previous versions?
Yeah, we can drop `trusted-host` for now. Most people have just reverted to pip 19.2.3
Thanks @surry for a well designed reproducer and @chrahunt for figuring out a potential root cause! :)

Resolving patch:
diff --git a/src/pip/_internal/network/session.py b/src/pip/_internal/network/session.py
--- a/src/pip/_internal/network/session.py
+++ b/src/pip/_internal/network/session.py
@@ -212,8 +212,9 @@ def close(self):
 class InsecureHTTPAdapter(HTTPAdapter):
 
     def cert_verify(self, conn, url, verify, cert):
-        conn.cert_reqs = 'CERT_NONE'
-        conn.ca_certs = None
+        super(InsecureHTTPAdapter, self).cert_verify(
+            conn=conn, url=url, verify=False, cert=cert
+        )
 
 
 class PipSession(requests.Session):


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -13,6 +13,7 @@
 
 from pip._internal.main import main as pip_entry_point
 from tests.lib import DATA_DIR, SRC_DIR, TestData
+from tests.lib.certs import make_tls_cert, serialize_cert, serialize_key
 from tests.lib.path import Path
 from tests.lib.scripttest import PipTestEnvironment
 from tests.lib.venv import VirtualEnvironment
@@ -385,3 +386,21 @@ def in_memory_pip():
 def deprecated_python():
     """Used to indicate whether pip deprecated this python version"""
     return sys.version_info[:2] in [(2, 7)]
+
+
+@pytest.fixture(scope="session")
+def cert_factory(tmpdir_factory):
+    def factory():
+        # type: () -> str
+        """Returns path to cert/key file.
+        """
+        output_path = Path(str(tmpdir_factory.mktemp("certs"))) / "cert.pem"
+        # Must be Text on PY2.
+        cert, key = make_tls_cert(u"localhost")
+        with open(str(output_path), "wb") as f:
+            f.write(serialize_cert(cert))
+            f.write(serialize_key(key))
+
+        return str(output_path)
+
+    return factory
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -2,6 +2,7 @@
 import glob
 import os
 import shutil
+import ssl
 import sys
 import textwrap
 from os.path import curdir, join, pardir
@@ -30,6 +31,12 @@
 from tests.lib.filesystem import make_socket_file
 from tests.lib.local_repos import local_checkout
 from tests.lib.path import Path
+from tests.lib.server import (
+    file_response,
+    make_mock_server,
+    package_page,
+    server_running,
+)
 
 skip_if_python2 = pytest.mark.skipif(PY2, reason="Non-Python 2 only")
 skip_if_not_python2 = pytest.mark.skipif(not PY2, reason="Python 2 only")
@@ -1729,3 +1736,39 @@ def test_install_yanked_file_and_print_warning(script, data):
     assert expected_warning in result.stderr, str(result)
     # Make sure a "yanked" release is installed
     assert 'Successfully installed simple-3.0\n' in result.stdout, str(result)
+
+
+@pytest.mark.parametrize("install_args", [
+    (),
+    ("--trusted-host", "localhost"),
+])
+def test_install_sends_client_cert(install_args, script, cert_factory, data):
+    cert_path = cert_factory()
+    ctx = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
+    ctx.load_cert_chain(cert_path, cert_path)
+    ctx.load_verify_locations(cafile=cert_path)
+    ctx.verify_mode = ssl.CERT_REQUIRED
+
+    server = make_mock_server(ssl_context=ctx)
+    server.mock.side_effect = [
+        package_page({
+            "simple-3.0.tar.gz": "/files/simple-3.0.tar.gz",
+        }),
+        file_response(str(data.packages / "simple-3.0.tar.gz")),
+    ]
+
+    url = "https://{}:{}/simple".format(server.host, server.port)
+
+    args = ["install", "-vvv", "--cert", cert_path, "--client-cert", cert_path]
+    args.extend(["--index-url", url])
+    args.extend(install_args)
+    args.append("simple")
+
+    with server_running(server):
+        script.pip(*args)
+
+    assert server.mock.call_count == 2
+    for call_args in server.mock.call_args_list:
+        environ, _ = call_args.args
+        assert "SSL_CLIENT_CERT" in environ
+        assert environ["SSL_CLIENT_CERT"]
diff --git a/tests/lib/certs.py b/tests/lib/certs.py
new file mode 100644
--- /dev/null
+++ b/tests/lib/certs.py
@@ -0,0 +1,53 @@
+from datetime import datetime, timedelta
+
+from cryptography import x509
+from cryptography.hazmat.backends import default_backend
+from cryptography.hazmat.primitives import hashes, serialization
+from cryptography.hazmat.primitives.asymmetric import rsa
+from cryptography.x509.oid import NameOID
+
+from pip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+    from typing import Text, Tuple
+
+
+def make_tls_cert(hostname):
+    # type: (Text) -> Tuple[x509.Certificate, rsa.RSAPrivateKey]
+    key = rsa.generate_private_key(
+        public_exponent=65537,
+        key_size=2048,
+        backend=default_backend()
+    )
+    subject = issuer = x509.Name([
+        x509.NameAttribute(NameOID.COMMON_NAME, hostname),
+    ])
+    cert = (
+        x509.CertificateBuilder()
+        .subject_name(subject)
+        .issuer_name(issuer)
+        .public_key(key.public_key())
+        .serial_number(x509.random_serial_number())
+        .not_valid_before(datetime.utcnow())
+        .not_valid_after(datetime.utcnow() + timedelta(days=10))
+        .add_extension(
+            x509.SubjectAlternativeName([x509.DNSName(hostname)]),
+            critical=False,
+        )
+        .sign(key, hashes.SHA256(), default_backend())
+    )
+    return cert, key
+
+
+def serialize_key(key):
+    # type: (rsa.RSAPrivateKey) -> bytes
+    return key.private_bytes(
+        encoding=serialization.Encoding.PEM,
+        format=serialization.PrivateFormat.TraditionalOpenSSL,
+        encryption_algorithm=serialization.NoEncryption(),
+    )
+
+
+def serialize_cert(cert):
+    # type: (x509.Certificate) -> bytes
+    return cert.public_bytes(serialization.Encoding.PEM)
diff --git a/tests/lib/server.py b/tests/lib/server.py
new file mode 100644
--- /dev/null
+++ b/tests/lib/server.py
@@ -0,0 +1,181 @@
+import os
+import signal
+import ssl
+import threading
+from contextlib import contextmanager
+from textwrap import dedent
+
+from mock import Mock
+from pip._vendor.contextlib2 import nullcontext
+from werkzeug.serving import WSGIRequestHandler
+from werkzeug.serving import make_server as _make_server
+
+from pip._internal.utils.typing import MYPY_CHECK_RUNNING
+
+if MYPY_CHECK_RUNNING:
+    from types import TracebackType
+    from typing import (
+        Any, Callable, Dict, Iterable, List, Optional, Text, Tuple, Type, Union
+    )
+
+    from werkzeug.serving import BaseWSGIServer
+
+    Environ = Dict[str, str]
+    Status = str
+    Headers = Iterable[Tuple[str, str]]
+    ExcInfo = Tuple[Type[BaseException], BaseException, TracebackType]
+    Write = Callable[[bytes], None]
+    StartResponse = Callable[[Status, Headers, Optional[ExcInfo]], Write]
+    Body = List[bytes]
+    Responder = Callable[[Environ, StartResponse], Body]
+
+    class MockServer(BaseWSGIServer):
+        mock = Mock()  # type: Mock
+
+# Applies on Python 2 and Windows.
+if not hasattr(signal, "pthread_sigmask"):
+    # We're not relying on this behavior anywhere currently, it's just best
+    # practice.
+    blocked_signals = nullcontext
+else:
+    @contextmanager
+    def blocked_signals():
+        """Block all signals for e.g. starting a worker thread.
+        """
+        old_mask = signal.pthread_sigmask(
+            signal.SIG_SETMASK, range(1, signal.NSIG)
+        )
+        try:
+            yield
+        finally:
+            signal.pthread_sigmask(signal.SIG_SETMASK, old_mask)
+
+
+class RequestHandler(WSGIRequestHandler):
+    def make_environ(self):
+        environ = super(RequestHandler, self).make_environ()
+
+        # From pallets/werkzeug#1469, will probably be in release after
+        # 0.16.0.
+        try:
+            # binary_form=False gives nicer information, but wouldn't be
+            # compatible with what Nginx or Apache could return.
+            peer_cert = self.connection.getpeercert(binary_form=True)
+            if peer_cert is not None:
+                # Nginx and Apache use PEM format.
+                environ["SSL_CLIENT_CERT"] = ssl.DER_cert_to_PEM_cert(
+                    peer_cert,
+                )
+        except ValueError:
+            # SSL handshake hasn't finished.
+            self.server.log("error", "Cannot fetch SSL peer certificate info")
+        except AttributeError:
+            # Not using TLS, the socket will not have getpeercert().
+            pass
+
+        return environ
+
+
+def mock_wsgi_adapter(mock):
+    # type: (Callable[[Environ, StartResponse], Responder]) -> Responder
+    """Uses a mock to record function arguments and provide
+    the actual function that should respond.
+    """
+    def adapter(environ, start_response):
+        # type: (Environ, StartResponse) -> Body
+        responder = mock(environ, start_response)
+        return responder(environ, start_response)
+
+    return adapter
+
+
+def make_mock_server(**kwargs):
+    # type: (Any) -> MockServer
+    kwargs.setdefault("request_handler", RequestHandler)
+
+    mock = Mock()
+    app = mock_wsgi_adapter(mock)
+    server = _make_server("localhost", 0, app=app, **kwargs)
+    server.mock = mock
+    return server
+
+
+@contextmanager
+def server_running(server):
+    # type: (BaseWSGIServer) -> None
+    thread = threading.Thread(target=server.serve_forever)
+    thread.daemon = True
+    with blocked_signals():
+        thread.start()
+    try:
+        yield
+    finally:
+        server.shutdown()
+        thread.join()
+
+
+# Helper functions for making responses in a declarative way.
+
+
+def text_html_response(text):
+    # type: (Text) -> Responder
+    def responder(environ, start_response):
+        # type: (Environ, StartResponse) -> Body
+        start_response("200 OK", [
+            ("Content-Type", "text/html; charset=UTF-8"),
+        ])
+        return [text.encode('utf-8')]
+
+    return responder
+
+
+def html5_page(text):
+    # type: (Union[Text, str]) -> Text
+    return dedent(u"""
+    <!DOCTYPE html>
+    <html>
+      <body>
+        {}
+      </body>
+    </html>
+    """).strip().format(text)
+
+
+def index_page(spec):
+    # type: (Dict[str, str]) -> Responder
+    def link(name, value):
+        return '<a href="{}">{}</a>'.format(
+            value, name
+        )
+
+    links = ''.join(link(*kv) for kv in spec.items())
+    return text_html_response(html5_page(links))
+
+
+def package_page(spec):
+    # type: (Dict[str, str]) -> Responder
+    def link(name, value):
+        return '<a href="{}">{}</a>'.format(
+            value, name
+        )
+
+    links = ''.join(link(*kv) for kv in spec.items())
+    return text_html_response(html5_page(links))
+
+
+def file_response(path):
+    # type: (str) -> Responder
+    def responder(environ, start_response):
+        # type: (Environ, StartResponse) -> Body
+        size = os.stat(path).st_size
+        start_response(
+            "200 OK", [
+                ("Content-Type", "application/octet-stream"),
+                ("Content-Length", str(size)),
+            ],
+        )
+
+        with open(path, 'rb') as f:
+            return [f.read()]
+
+    return responder
diff --git a/tools/requirements/tests.txt b/tools/requirements/tests.txt
--- a/tools/requirements/tests.txt
+++ b/tools/requirements/tests.txt
@@ -1,3 +1,4 @@
+cryptography==2.8
 freezegun
 mock
 pretend
@@ -12,4 +13,5 @@ pyyaml
 setuptools>=39.2.0  # Needed for `setuptools.wheel.Wheel` support.
 scripttest
 https://github.com/pypa/virtualenv/archive/master.zip#egg=virtualenv
+werkzeug==0.16.0
 wheel