Repository: python-poetry/poetry

Issue:
Some PyPI dependencies not installed when installing a package from a private repository
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **OS version and name**: Ubuntu 19.10
- **Poetry version**: 1.0.2

## Issue
<!-- Now feel free to write your issue, but please be descriptive! Thanks again 🙌 ❤️ -->
When running ``poetry install`` with the ``pyproject.toml`` containing a package from a private repository, poetry does install most of the packages from PyPI, but when it comes to installing the private package itself, it fails to install some dependencies:
```
 Looking in indexes: https://private-repository-url/+simple, https://pypi.org/
 Collecting poetry>=0.12
   ERROR: Could not find a version that satisfies the requirement poetry>=0.12 (from versions: none)

```
The workaround was to specify the PyPI source in ``pyproject.toml``:
```
[[tool.poetry.source]]
name = "pypi-repository"
url = "https://pypi.org/simple/"
default = true
```

So it seems that a possible fix is to add ``simple/`` at the end of PyPI url.


Discussion:
Hello @vladimir-genkin ,

what you describe is exactly this what is written in the docs: https://python-poetry.org/docs/repositories/#install-dependencies-from-a-private-repository

fin swimmer
Hey guys, we just spent 3 hours debugging this issue. @finswimmer why not auto-append /simple from the end of the URL if it's missing?
See near-duplicate: #1942 
@finswimmer the docs say that the private repository will take precedence over PyPI, but will not be used exclusively unless that source is configured as the default source. The bug here (and in #1942) is that when a package is installed from a private repository, dependencies of the private package that are on PyPI are not found (even though the private repository is not configured to be the default) because the PyPI URL that poetry passes to pip is incorrect.

To demonstrate that this is the case, I added `secondary = true` to the source definition for our private repository in our `pyproject.toml`, exactly as recommended by the docs you linked to, but poetry still cannot find setuptools when trying to install a direct dependency from our private repository:

```
[EnvCommandError]
Command ['/Users/Daniel/src/work/redacted/data-atlas/env/bin/pip', 'install', '--no-deps', '--cert', '/Users/Daniel/.pip/certs/PythonIndicesCerts.pem', '--index-url', 'https://redacted.com/api/pypi/redacted-pypi/simple', '--extra-index-url', 'https://pypi.org/',
 'fucine==0.2.0'] errored with the following return code 1, and output:
Looking in indexes: https://redacted.com/api/pypi/datsci-pypi/simple, https://pypi.org/
Collecting fucine==0.2.0
  Using cached https://redacted.com/api/pypi/datsci-pypi/fucine/0.2.0/fucine-0.2.0.tar.gz
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
  Complete output from command /Users/Daniel/src/work/redacted/data-atlas/env/bin/python3.6 -m pip install --ignore-installed --no-user --prefix /private/var/folders/hc/c5w3vvjs221c7bbb50m9lyn40000gp/T/pip-build-env-qpmwx5_4 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://redacted.com/api/pypi/datsci-pypi/simple --extra-index-url https://pypi.org/ -- setuptools wheel:
  Looking in indexes: https://redacted.com/api/pypi/datsci-pypi/simple, https://pypi.org/
  Collecting setuptools
    Could not find a version that satisfies the requirement setuptools (from versions: )
```
Note the incorrect PyPI URL `https://pypi.org/` appears throughout, instead of the correct URL `https://pypi.org/simple/`.

Adding PyPI explicitly to our `pyproject.toml` as a poetry source, with the correct URL `https://pypi.org/simple/`, resolves the problem, as it did for @vladimir-genkin.

Resolving patch:
diff --git a/poetry/repositories/pypi_repository.py b/poetry/repositories/pypi_repository.py
--- a/poetry/repositories/pypi_repository.py
+++ b/poetry/repositories/pypi_repository.py
@@ -32,7 +32,7 @@
 from poetry.version.markers import parse_marker
 
 from .exceptions import PackageNotFound
-from .repository import Repository
+from .remote_repository import RemoteRepository
 
 
 try:
@@ -46,12 +46,14 @@
 logger = logging.getLogger(__name__)
 
 
-class PyPiRepository(Repository):
+class PyPiRepository(RemoteRepository):
 
     CACHE_VERSION = parse_constraint("1.0.0")
 
     def __init__(self, url="https://pypi.org/", disable_cache=False, fallback=True):
-        self._url = url
+        super(PyPiRepository, self).__init__(url.rstrip("/") + "/simple/")
+
+        self._base_url = url
         self._disable_cache = disable_cache
         self._fallback = fallback
 
@@ -71,18 +73,8 @@ def __init__(self, url="https://pypi.org/", disable_cache=False, fallback=True):
         self._session = CacheControl(session(), cache=self._cache_control_cache)
         self._inspector = Inspector()
 
-        super(PyPiRepository, self).__init__()
-
         self._name = "PyPI"
 
-    @property
-    def url(self):  # type: () -> str
-        return self._url
-
-    @property
-    def authenticated_url(self):  # type: () -> str
-        return self._url
-
     def find_packages(
         self,
         name,  # type: str
@@ -224,7 +216,7 @@ def search(self, query):
 
         search = {"q": query}
 
-        response = session().get(self._url + "search", params=search)
+        response = session().get(self._base_url + "search", params=search)
         content = parse(response.content, namespaceHTMLElements=False)
         for result in content.findall(".//*[@class='package-snippet']"):
             name = result.find("h3/*[@class='package-snippet__name']").text
@@ -361,12 +353,12 @@ def _get_release_info(self, name, version):  # type: (str, str) -> dict
 
     def _get(self, endpoint):  # type: (str) -> Union[dict, None]
         try:
-            json_response = self._session.get(self._url + endpoint)
+            json_response = self._session.get(self._base_url + endpoint)
         except TooManyRedirects:
             # Cache control redirect loop.
             # We try to remove the cache and try again
-            self._cache_control_cache.delete(self._url + endpoint)
-            json_response = self._session.get(self._url + endpoint)
+            self._cache_control_cache.delete(self._base_url + endpoint)
+            json_response = self._session.get(self._base_url + endpoint)
 
         if json_response.status_code == 404:
             return None
diff --git a/poetry/repositories/remote_repository.py b/poetry/repositories/remote_repository.py
new file mode 100644
--- /dev/null
+++ b/poetry/repositories/remote_repository.py
@@ -0,0 +1,16 @@
+from .repository import Repository
+
+
+class RemoteRepository(Repository):
+    def __init__(self, url):  # type: (str) -> None
+        self._url = url
+
+        super(RemoteRepository, self).__init__()
+
+    @property
+    def url(self):  # type: () -> str
+        return self._url
+
+    @property
+    def authenticated_url(self):  # type: () -> str
+        return self._url


Test patch:
diff --git a/tests/repositories/test_pypi_repository.py b/tests/repositories/test_pypi_repository.py
--- a/tests/repositories/test_pypi_repository.py
+++ b/tests/repositories/test_pypi_repository.py
@@ -208,3 +208,10 @@ def test_get_should_invalid_cache_on_too_many_redirects_error(mocker):
     repository._get("https://pypi.org/pypi/async-timeout/json")
 
     assert delete_cache.called
+
+
+def test_urls():
+    repository = PyPiRepository()
+
+    assert "https://pypi.org/simple/" == repository.url
+    assert "https://pypi.org/simple/" == repository.authenticated_url