Repository: python-poetry/poetry

Issue:
Upgrading packages from private repositories fails using cert and client-cert 
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **Poetry version**: **1.1.4** <!-- Replace with version -->

## Issue
<!-- Now feel free to write your issue, but please be descriptive! Thanks again 🙌 ❤️ -->

To configure the cert and client-cert I am using:

    poetry config certificates.private-pypi.cert /path/to/ca.crt
    poetry config certificates.private-pypi.client-cert /path/to/client.pem

... from the custom CA & TLS [instructions](https://python-poetry.org/docs/repositories/#custom-certificate-authority-and-mutual-tls-authentication).

And from pyproject.toml:

    [[tool.poetry.source]]
    name = "private-pypi"
    url = "https://private-pypi/simple"

When doing a `poetry update` to get the latest packages, it fails to get packages from private repositories. 

It fails with:

    Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify
     failed: self signed certificate in certificate chain (_ssl.c:1091)'))

... I started to debug that error it looked like the cert and CA were simply not provided in the http request. Actually here's what I saw:

A single successful request where the cert and CA were passed:

    GET https://private-pypi/simple/<package_name>/

And then 5 failed retries that had the SSLCertVerificationError:

    GET https://private-pypi/packages/<package_name>/<package>-py3-none-any.whl#md5=blah

And in those 5 requests it looked like the cert and CA were not provided. So in my CI build I am not able to install `<package>` which happens to be an upgrade. I believe this started to happen with 1.1.0 as I can force version 1.0.10 and then the CI build will succeed and the package will install successfully.


Discussion:
Yes, it's because the certs aren't provided to the second request which actually downloads the archive. I'm posting a pull-request to fix this this morning.
Additional context:

The `_download_archive` method doesn't supply certs for the config. Can verify that the issue is relieved by adding a `verify` keyword to the request

See lib/poetry/installation/executor.py:616:
```
    def _download_archive(self, operation, link):  # type: (Operation, Link) -> Path
        response = self._authenticator.request(
            "get", link.url, stream=True, io=self._sections.get(id(operation), self._io),
            # verify="ca.crt",  # If I add this in, the request functions fine
        )
        wheel_size = response.headers.get("content-length")
        operation_message = self.get_operation_message(operation)
        message = "  <fg=blue;options=bold>•</> {message}: <info>Downloading...</>".format(
           message=operation_message,
        )
```

Without it, I get errors like this:
```
  • Installing fetch-config (0.4.8): Pending...
Retrying HTTP request in 0.5 seconds.
Retrying HTTP request in 1.0 seconds.
Retrying HTTP request in 1.5 seconds.
Retrying HTTP request in 2.0 seconds.
  • Installing fetch-config (0.4.8): Failed

  SSLError

  HTTPSConnectionPool(host='---redacted---', port=443): Max retries exceeded with url: /api/v4/projects/1275/packages/pypi/files/fd1d8885b89b3586dfcb7266c5880e06836dcb09a6265a90cc55cf2267f2daa2/fetch_config-0.4.8-py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1108)')))
```

+1 for me, experiencing Unknown CA in packet data followed by client FIN.
Poetry version 1.1.4

This is in regards specifically to `poetry publish` and is blocking our adoption of this tool.

```
No suitable keyring backend found
No suitable keyring backends were found
Using a plaintext file to store and retrieve credentials
Publishing poetry-demo (0.1.1) to dev_artifactory
 - Uploading poetry-demo-0.1.1.tar.gz FAILED

  Stack trace:

  9  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/console_application.py:131 in run
      129│             parsed_args = resolved_command.args
      130│
    → 131│             status_code = command.handle(parsed_args, io)
      132│         except KeyboardInterrupt:
      133│             status_code = 1

  8  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/api/command/command.py:120 in handle
      118│     def handle(self, args, io):  # type: (Args, IO) -> int
      119│         try:
    → 120│             status_code = self._do_handle(args, io)
      121│         except KeyboardInterrupt:
      122│             if io.is_debug():

  7  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/clikit/api/command/command.py:171 in _do_handle
      169│         handler_method = self._config.handler_method
      170│
    → 171│         return getattr(handler, handler_method)(args, io, self)
      172│
      173│     def __repr__(self):  # type: () -> str

  6  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/cleo/commands/command.py:92 in wrap_handle
       90│         self._command = command
       91│
    →  92│         return self.handle()
       93│
       94│     def handle(self):  # type: () -> Optional[int]

  5  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/console/commands/publish.py:77 in handle
      75│         )
      76│
    → 77│         publisher.publish(
      78│             self.option("repository"),
      79│             self.option("username"),

  4  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/publisher.py:93 in publish
      91│         )
      92│
    → 93│         self._uploader.upload(
      94│             url,
      95│             cert=cert or get_cert(self._poetry.config, repository_name),

  3  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:119 in upload
      117│
      118│         try:
    → 119│             self._upload(session, url, dry_run)
      120│         finally:
      121│             session.close()

  2  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:205 in _upload
      203│     ):  # type: (requests.Session, str, Optional[bool]) -> None
      204│         try:
    → 205│             self._do_upload(session, url, dry_run)
      206│         except HTTPError as e:
      207│             if (

  1  ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:224 in _do_upload
      222│             # TODO: Check existence
      223│
    → 224│             resp = self._upload_file(session, url, file, dry_run)
      225│
      226│             if not dry_run:

  UploadError

  Connection Error: We were unable to connect to the repository, ensure the url is correct and can be reached.

  at ~/.local/pipx/venvs/poetry/lib/python3.8/site-packages/poetry/publishing/uploader.py:282 in _upload_file
      278│                         " - Uploading {0} {1}".format(
      279│                             file.name, "FAILED"
      280│                         )
      281│                     )
    → 282│                 raise UploadError(e)
      283│             finally:
      284│                 self._io.write_line("")
      285│
      286│         return resp
```

![image](https://user-images.githubusercontent.com/6843296/104789844-b4afe980-575b-11eb-8f1c-4eb91ad50ec5.png)

With the addition of a patch I supplied in the PR 3490's comments, this seems to fix the issue that we're experiencing when attempting to download an archive from a private pypi repo that requires client certs for all accesses.
Also seeing this issue, cant post anymore info but we are able to pull from private repo using certs and using `pip install` but poetry fails cert error using same certificates and token as used in the pip install.
Any updates on this? Currently blocking our project from upgrading poetry past 1.0.10

Resolving patch:
diff --git a/src/poetry/utils/authenticator.py b/src/poetry/utils/authenticator.py
--- a/src/poetry/utils/authenticator.py
+++ b/src/poetry/utils/authenticator.py
@@ -6,16 +6,21 @@
 
 from typing import TYPE_CHECKING
 from typing import Any
+from typing import Iterator
 
 import requests
 import requests.auth
 import requests.exceptions
 
 from poetry.exceptions import PoetryException
+from poetry.utils.helpers import get_cert
+from poetry.utils.helpers import get_client_cert
 from poetry.utils.password_manager import PasswordManager
 
 
 if TYPE_CHECKING:
+    from pathlib import Path
+
     from cleo.io.io import IO
 
     from poetry.config.config import Config
@@ -30,6 +35,7 @@ def __init__(self, config: Config, io: IO | None = None) -> None:
         self._io = io
         self._session = None
         self._credentials = {}
+        self._certs = {}
         self._password_manager = PasswordManager(self._config)
 
     def _log(self, message: str, level: str = "debug") -> None:
@@ -61,8 +67,16 @@ def request(self, method: str, url: str, **kwargs: Any) -> requests.Response:
 
         proxies = kwargs.get("proxies", {})
         stream = kwargs.get("stream")
-        verify = kwargs.get("verify")
-        cert = kwargs.get("cert")
+
+        certs = self.get_certs_for_url(url)
+        verify = kwargs.get("verify") or certs.get("verify")
+        cert = kwargs.get("cert") or certs.get("cert")
+
+        if cert is not None:
+            cert = str(cert)
+
+        if verify is not None:
+            verify = str(verify)
 
         settings = session.merge_environment_settings(
             prepared_request.url, proxies, stream, verify, cert
@@ -157,7 +171,7 @@ def _get_http_auth(self, name: str, netloc: str | None) -> dict[str, str] | None
             return auth
 
     def _get_credentials_for_netloc(self, netloc: str) -> tuple[str | None, str | None]:
-        for repository_name in self._config.get("repositories", []):
+        for repository_name, _ in self._get_repository_netlocs():
             auth = self._get_http_auth(repository_name, netloc)
 
             if auth is None:
@@ -167,6 +181,22 @@ def _get_credentials_for_netloc(self, netloc: str) -> tuple[str | None, str | No
 
         return None, None
 
+    def get_certs_for_url(self, url: str) -> dict[str, Path | None]:
+        parsed_url = urllib.parse.urlsplit(url)
+
+        netloc = parsed_url.netloc
+
+        return self._certs.setdefault(
+            netloc,
+            self._get_certs_for_netloc_from_config(netloc),
+        )
+
+    def _get_repository_netlocs(self) -> Iterator[tuple[str, str]]:
+        for repository_name in self._config.get("repositories", []):
+            url = self._config.get(f"repositories.{repository_name}.url")
+            parsed_url = urllib.parse.urlsplit(url)
+            yield repository_name, parsed_url.netloc
+
     def _get_credentials_for_netloc_from_keyring(
         self, url: str, netloc: str, username: str | None
     ) -> dict[str, str] | None:
@@ -193,3 +223,14 @@ def _get_credentials_for_netloc_from_keyring(
             }
 
         return None
+
+    def _get_certs_for_netloc_from_config(self, netloc: str) -> dict[str, Path | None]:
+        certs = {"cert": None, "verify": None}
+
+        for repository_name, repository_netloc in self._get_repository_netlocs():
+            if netloc == repository_netloc:
+                certs["cert"] = get_client_cert(self._config, repository_name)
+                certs["verify"] = get_cert(self._config, repository_name)
+                break
+
+        return certs


Test patch:
diff --git a/tests/utils/test_authenticator.py b/tests/utils/test_authenticator.py
--- a/tests/utils/test_authenticator.py
+++ b/tests/utils/test_authenticator.py
@@ -4,6 +4,7 @@
 import uuid
 
 from dataclasses import dataclass
+from pathlib import Path
 from typing import TYPE_CHECKING
 from typing import Any
 
@@ -306,3 +307,46 @@ def test_authenticator_uses_env_provided_credentials(
     request = http.last_request()
 
     assert request.headers["Authorization"] == "Basic YmFyOmJheg=="
+
+
+@pytest.mark.parametrize(
+    "cert,client_cert",
+    [
+        (None, None),
+        (None, "path/to/provided/client-cert"),
+        ("/path/to/provided/cert", None),
+        ("/path/to/provided/cert", "path/to/provided/client-cert"),
+    ],
+)
+def test_authenticator_uses_certs_from_config_if_not_provided(
+    config: Config,
+    mock_remote: type[httpretty.httpretty],
+    http: type[httpretty.httpretty],
+    mocker: MockerFixture,
+    cert: str | None,
+    client_cert: str | None,
+):
+    configured_cert = "/path/to/cert"
+    configured_client_cert = "/path/to/client-cert"
+    config.merge(
+        {
+            "repositories": {"foo": {"url": "https://foo.bar/simple/"}},
+            "http-basic": {"foo": {"username": "bar", "password": "baz"}},
+            "certificates": {
+                "foo": {"cert": configured_cert, "client-cert": configured_client_cert}
+            },
+        }
+    )
+
+    authenticator = Authenticator(config, NullIO())
+    session_send = mocker.patch.object(authenticator.session, "send")
+    authenticator.request(
+        "get",
+        "https://foo.bar/files/foo-0.1.0.tar.gz",
+        verify=cert,
+        cert=client_cert,
+    )
+    kwargs = session_send.call_args[1]
+
+    assert Path(kwargs["verify"]) == Path(cert or configured_cert)
+    assert Path(kwargs["cert"]) == Path(client_cert or configured_client_cert)