Repository: python-poetry/poetry

Issue:
Poetry not respecting order of evaluation for repository sources
- [X] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [X] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.

- **OS version and name**: Mac OS 10.15.3
- **Poetry version**: 1.0.5

## Issue
So, I am trying to configure a pyproject.toml file that supports pulling files from pypi.org whenever possible (ie: because on my site the performance is better) but will support pulling packages that aren't found on pypi.org from a secondary location (ie: a private pypi repo). So to start with I added a section like what follows to a sample toml file for testing:

```
[[tool.poetry.source]]
name = 'default'
url = 'https://pypi.python.org/simple'
default = true

[[tool.poetry.source]]
name = "private"
url = "https://private/server/url
secondary = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"
```

Based on the docs, I was under the impression that by putting `default=true` on the first repo config, and `secondary=true` on the other one, that my goal would be achieved: if a package exists on pypi.org it'd pull it from there and if not it'd pull it from my secondary repo. However, that does not seem to be the case.

For the sake of this discussion I am using average performance metrics from running `poetry lock` on this toml file and comparing the length of time it takes to resolve the dependencies of the one Python package mentioned (ie: `sphinx` which is a standard Python package available on pypi.org). 

So, running `poetry lock` on this file using the configuration I posted above takes about 20-25 seconds to complete. For comparison purposes I simply removed the second source definition from the toml file giving me:

```
[[tool.poetry.source]]
name = 'default'
url = 'https://pypi.python.org/simple'
default = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"
```

Re-running the same common against this toml file takes between 2 and 3 seconds to complete - a full order of magnitude difference in performance. Now, for my 3rd and final test I removed the first repo source leaving just the second one, giving me the following configuration block:

```
[[tool.poetry.source]]
name = "private"
url = "https://private/server/url
secondary = true

[tool.poetry.dependencies]
python = "^3.6"
sphinx = "*"
```

Re-running the same command again takes about the same 20-25 seconds to complete (NOTE: our private pypi repository is also a mirror of the public pypi.org repo so I can compare the same package versions and resolution times in all cases).

So, based on these results, the only explanation I can see is that when the second repository definition is in place the public pypi.org repo is being ignored or something and the secondary / private repo is still being accessed for some reason.

My expectation here is that poetry would iterate over the various repos defined in the toml file in order of declaration. For each package listed in the toml file it should then try to access / index each package against each repository stopping when it finds the first match. If this were true then I would have expected my first test case above to perform identically to the second one because the package I'm testing (sphinx) should exist on pypi.org, and all of it's transitive dependencies will be available there as well. However this is obviously not the case.

So my question is - is this a bug or is there some additional configuration options I need to specify to get this to work as I was expecting?
Custom repository doesn't have precedence over PyPI
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **OS version and name**: macOS Mojave 10.14.6
- **Poetry version**: 1.1.3
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**:
```toml
[[tool.poetry.source]]
name = 'foobar'
url = 'http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple'

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```

## Issue
In docs we can read:
> Any custom repository will have precedence over PyPI.

In the above example, `foobar` is a proxy repo incl. both internal packages and the public ones from PyPI. Running `poetry install` should cause a creation of `poetry.lock`, where all the packages have `foobar` in their `source` field. However, it's completely opposite - none of the packages has a `source` field, which means all of them are downloaded directly from PyPI.

When I add `default = true` to the above `source` element, which means PyPI is disabled, everything works properly - generated `poetry.lock` includes `foobar` as a `source` for all packages and installation doesn't break, so it's not a problem with repository itself.

It's especially frustrating when you want to use `poetry export` (for any reason), because then the final `requirements.txt` doesn't include neither `--index-url` nor `--extra-index-url` (it seems it depends on the actually used sources, not the `pyproject.toml` content).


Discussion:
For kicks I just tried to run a few similar / comparable tests using `pip` directly (which poetry makes use of under the hood), making use of the index-url and extra-index-url parameters and I seem to be getting very similar results. Making the main index URL my private repo is slow, as is making my private repo an "extra" index and leaving the pypi.org as the default index URL, but when I use pypi.org exclusively the performance is better.

So maybe this is some odd limitation of pip and it just so happens that poetry is sharing that functionality.

I wonder if this is something new that has changed in a recent version of pip / poetry or something. I am pretty sure I've tried scenarios like this in the past with different results. I don't have detailed metrics from those days-gone-past though so I can't say for sure. :(

Still - any input anyone might have on whether there is anything that can be done to help mitigate this problem would be appreciated.
NOTE: if it helps, I could create a map / list of packages that I know are exclusively only available on my private pypi server and thus could associate those packages with my mirror, and all others to the main pypi.org site. Not sure if that is possible or not though.
I'm seeing the same (using a private repo as a secondary source) and am scratching my head at the following output:

```
> poetry update -vvv
Using virtualenv: C:\[...]\.venv
Updating dependencies
Resolving dependencies...
   1: fact: XXX is 0.39.5
   1: derived: XXX
   1: fact: XXX depends on pyparsing (^2.2)
[...]
   1: fact: XXX depends on rope (^0.17.0)
   1: selecting XXX (0.39.5)
   1: derived: rope (^0.17.0)
[...]
   1: derived: YYY (^1.32)
PyPI: No packages found for YYY >=1.32,<2.0
dama: https://PRIVATE_PYPI/simple/YYY/
dama: 52 packages found for YYY >=1.32,<2.0
PyPI: No release information found for rope-0.2, skipping
[...]]
PyPI: No release information found for rope-0.9, skipping
PyPI: No release information found for rope-0.9.1, skipping
PyPI: 1 packages found for rope >=0.17.0,<0.18.0
dama: https://PRIVATE_PYPI/simple/rope/
   1: Version solving took 1.902 seconds.
   1: Tried 1 solutions.

[Crash because our private PyPI does not forward to the official one using https]
```

The last lines seem to implicate that `rope` is found on the official PyPI, but then poetry goes off and queries the private repo for it anyway?!
I face same issue here, it seems to be, that poetry/pip checks packages for both default and secondary sources. It makes sense if we are talking about robustness - you'll always get version that will fit you best from poetry/pip point of view, but even so I would suggest at least some key/argument to avoid this deep check in case package fit the dependencies already found during search in some previous repo.
I've just found the very similar issues:
https://github.com/python-poetry/poetry/issues/1677
https://github.com/python-poetry/poetry/issues/2564
but the problem seems not to be resolved for almost a year, so I will keep this issue open to bump the topic.
Can you please try the fix at #3251.

### Using pipx
```sh
pipx install --suffix=@3251 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head'
```

### Using a container (podman | docker)
```sh
podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head
install -d foobar
pushd foobar
cat > pyproject.toml <<TOML
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Your Name <your.name@example.com>"]

[[tool.poetry.source]]
name = 'testpypi'
url = 'https://test.pypi.org/simple'

[tool.poetry.dependencies]
python = "^3.8"
py-serializer = "0.9.5"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
TOML
poetry lock
poetry export -f requirements.txt
EOF
```
is this the same issue as #3249 ?
@abn I've jus tested the PR. `poetry.lock` seems good, there's a proper `[package.source]` table for all packages:
```toml
[package.source]
type = "legacy"
url = "http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple"
reference = 'foobar'
```
But... there's one more thing. I've also run:
`poetry@3251 export -f requirements.txt -o requirements.txt --without-hashes --with-credentials`
and the first line of the output file is:
`--extra-index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple`

When using `default = true` we get then:
`--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple`

At first sight it seems okay, but we have to remember about the issue with pip and `--extra-index-url`:
https://pydist.com/blog/extra-index-url
https://github.com/pypa/pip/issues/5045
I assume it's already handled by Poetry itself, but to make it error-prone when exporting the `requirements.txt` file imho we should also keep the approach of specifying individual index per each package, e.g.:
`--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple anyconfig==0.9.11`

PS Of course there should be also `--trusted-host` flag in any of these scenarios due to `http`, but there was another PR for that, afair - already merged, just not released yet.
@jaklan appreciate the input on that. I'd suggest we raise that as a new issue. I agree that a per package index is better for the extra-index case. One worry, however, is that people seem to be attached to the current export format because some do text processing of the output it for various reasons.
@abn here it is: #3269.
Resolved with `1.1.4`.
There is a regression in `1.1.4` caused by the above fix:
https://github.com/python-poetry/poetry/issues/3306

Resolving patch:
diff --git a/poetry/factory.py b/poetry/factory.py
--- a/poetry/factory.py
+++ b/poetry/factory.py
@@ -153,14 +153,15 @@ def configure_sources(
 
             poetry.pool.add_repository(repository, is_default, secondary=is_secondary)
 
-        # Always put PyPI last to prefer private repositories
-        # but only if we have no other default source
-        if not poetry.pool.has_default():
-            has_sources = bool(sources)
-            poetry.pool.add_repository(PyPiRepository(), not has_sources, has_sources)
-        else:
+        # Put PyPI last to prefer private repositories
+        # unless we have no default source AND no primary sources
+        # (default = false, secondary = false)
+        if poetry.pool.has_default():
             if io.is_debug():
                 io.write_line("Deactivating the PyPI repository")
+        else:
+            default = not poetry.pool.has_primary_repositories()
+            poetry.pool.add_repository(PyPiRepository(), default, not default)
 
     @classmethod
     def create_legacy_repository(
diff --git a/poetry/repositories/pool.py b/poetry/repositories/pool.py
--- a/poetry/repositories/pool.py
+++ b/poetry/repositories/pool.py
@@ -25,6 +25,7 @@ def __init__(
         self._lookup: Dict[str, int] = {}
         self._repositories: List[Repository] = []
         self._default = False
+        self._has_primary_repositories = False
         self._secondary_start_idx = None
 
         for repository in repositories:
@@ -41,6 +42,9 @@ def repositories(self) -> List[Repository]:
     def has_default(self) -> bool:
         return self._default
 
+    def has_primary_repositories(self) -> bool:
+        return self._has_primary_repositories
+
     def has_repository(self, name: str) -> bool:
         name = name.lower() if name is not None else None
 
@@ -84,6 +88,7 @@ def add_repository(
             self._repositories.append(repository)
             self._lookup[repository_name] = len(self._repositories) - 1
         else:
+            self._has_primary_repositories = True
             if self._secondary_start_idx is None:
                 self._repositories.append(repository)
                 self._lookup[repository_name] = len(self._repositories) - 1


Test patch:
diff --git a/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml b/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml
@@ -0,0 +1,24 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
+
+[[tool.poetry.source]]
+name = "bar"
+url = "https://bar.baz/simple/"
+secondary = true
diff --git a/tests/fixtures/with_non_default_multiple_sources/pyproject.toml b/tests/fixtures/with_non_default_multiple_sources/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_multiple_sources/pyproject.toml
@@ -0,0 +1,23 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
+
+[[tool.poetry.source]]
+name = "bar"
+url = "https://bar.baz/simple/"
diff --git a/tests/fixtures/with_non_default_secondary_source/pyproject.toml b/tests/fixtures/with_non_default_secondary_source/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_secondary_source/pyproject.toml
@@ -0,0 +1,19 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
diff --git a/tests/test_factory.py b/tests/test_factory.py
--- a/tests/test_factory.py
+++ b/tests/test_factory.py
@@ -176,6 +176,64 @@ def test_poetry_with_non_default_source():
     assert isinstance(poetry.pool.repositories[1], PyPiRepository)
 
 
+def test_poetry_with_non_default_secondary_source():
+    poetry = Factory().create_poetry(fixtures_dir / "with_non_default_secondary_source")
+
+    assert len(poetry.pool.repositories) == 2
+
+    assert poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+
+def test_poetry_with_non_default_multiple_secondary_sources():
+    poetry = Factory().create_poetry(
+        fixtures_dir / "with_non_default_multiple_secondary_sources"
+    )
+
+    assert len(poetry.pool.repositories) == 3
+
+    assert poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[2]
+    assert repository.name == "bar"
+    assert isinstance(repository, LegacyRepository)
+
+
+def test_poetry_with_non_default_multiple_sources():
+    poetry = Factory().create_poetry(fixtures_dir / "with_non_default_multiple_sources")
+
+    assert len(poetry.pool.repositories) == 3
+
+    assert not poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "bar"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[2]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+
 def test_poetry_with_no_default_source():
     poetry = Factory().create_poetry(fixtures_dir / "sample_project")