Repository: python-poetry/poetry

Issue:
Custom repository doesn't have precedence over PyPI
<!--
  Hi there! Thank you for discovering and submitting an issue.

  Before you submit this; let's make sure of a few things.
  Please make sure the following boxes are ticked if they are correct.
  If not, please try and fulfill these first.
-->

<!-- Checked checkbox should look like this: [x] -->
- [x] I am on the [latest](https://github.com/python-poetry/poetry/releases/latest) Poetry version.
- [x] I have searched the [issues](https://github.com/python-poetry/poetry/issues) of this repo and believe that this is not a duplicate.
- [x] If an exception occurs when executing a command, I executed it again in debug mode (`-vvv` option).

<!--
  Once those are done, if you're able to fill in the following list with your information,
  it'd be very helpful to whoever handles the issue.
-->

- **OS version and name**: macOS Mojave 10.14.6
- **Poetry version**: 1.1.3
- **Link of a [Gist](https://gist.github.com/) with the contents of your pyproject.toml file**:
```toml
[[tool.poetry.source]]
name = 'foobar'
url = 'http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple'

[build-system]
requires = ["poetry_core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
```

## Issue
In docs we can read:
> Any custom repository will have precedence over PyPI.

In the above example, `foobar` is a proxy repo incl. both internal packages and the public ones from PyPI. Running `poetry install` should cause a creation of `poetry.lock`, where all the packages have `foobar` in their `source` field. However, it's completely opposite - none of the packages has a `source` field, which means all of them are downloaded directly from PyPI.

When I add `default = true` to the above `source` element, which means PyPI is disabled, everything works properly - generated `poetry.lock` includes `foobar` as a `source` for all packages and installation doesn't break, so it's not a problem with repository itself.

It's especially frustrating when you want to use `poetry export` (for any reason), because then the final `requirements.txt` doesn't include neither `--index-url` nor `--extra-index-url` (it seems it depends on the actually used sources, not the `pyproject.toml` content).


Discussion:
I've just found the very similar issues:
https://github.com/python-poetry/poetry/issues/1677
https://github.com/python-poetry/poetry/issues/2564
but the problem seems not to be resolved for almost a year, so I will keep this issue open to bump the topic.
Can you please try the fix at #3251.

### Using pipx
```sh
pipx install --suffix=@3251 'poetry @ git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head'
```

### Using a container (podman | docker)
```sh
podman run --rm -i --entrypoint bash python:3.8 <<EOF
set -xe
python -m pip install -q git+https://github.com/python-poetry/poetry.git@refs/pull/3251/head
install -d foobar
pushd foobar
cat > pyproject.toml <<TOML
[tool.poetry]
name = "foobar"
version = "0.1.0"
description = ""
authors = ["Your Name <your.name@example.com>"]

[[tool.poetry.source]]
name = 'testpypi'
url = 'https://test.pypi.org/simple'

[tool.poetry.dependencies]
python = "^3.8"
py-serializer = "0.9.5"

[tool.poetry.dev-dependencies]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"
TOML
poetry lock
poetry export -f requirements.txt
EOF
```
is this the same issue as #3249 ?
@abn I've jus tested the PR. `poetry.lock` seems good, there's a proper `[package.source]` table for all packages:
```toml
[package.source]
type = "legacy"
url = "http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple"
reference = 'foobar'
```
But... there's one more thing. I've also run:
`poetry@3251 export -f requirements.txt -o requirements.txt --without-hashes --with-credentials`
and the first line of the output file is:
`--extra-index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple`

When using `default = true` we get then:
`--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple`

At first sight it seems okay, but we have to remember about the issue with pip and `--extra-index-url`:
https://pydist.com/blog/extra-index-url
https://github.com/pypa/pip/issues/5045
I assume it's already handled by Poetry itself, but to make it error-prone when exporting the `requirements.txt` file imho we should also keep the approach of specifying individual index per each package, e.g.:
`--index-url http://repository.intranet.foobar.com/artifactory/api/pypi/pypi-virtual/simple anyconfig==0.9.11`

PS Of course there should be also `--trusted-host` flag in any of these scenarios due to `http`, but there was another PR for that, afair - already merged, just not released yet.
@jaklan appreciate the input on that. I'd suggest we raise that as a new issue. I agree that a per package index is better for the extra-index case. One worry, however, is that people seem to be attached to the current export format because some do text processing of the output it for various reasons.
@abn here it is: #3269.
Resolved with `1.1.4`.
There is a regression in `1.1.4` caused by the above fix:
https://github.com/python-poetry/poetry/issues/3306

Resolving patch:
diff --git a/poetry/factory.py b/poetry/factory.py
--- a/poetry/factory.py
+++ b/poetry/factory.py
@@ -88,14 +88,15 @@ def create_poetry(
 
             poetry.pool.add_repository(repository, is_default, secondary=is_secondary)
 
-        # Always put PyPI last to prefer private repositories
-        # but only if we have no other default source
-        if not poetry.pool.has_default():
-            has_sources = bool(sources)
-            poetry.pool.add_repository(PyPiRepository(), not has_sources, has_sources)
-        else:
+        # Put PyPI last to prefer private repositories
+        # unless we have no default source AND no primary sources
+        # (default = false, secondary = false)
+        if poetry.pool.has_default():
             if io.is_debug():
                 io.write_line("Deactivating the PyPI repository")
+        else:
+            default = not poetry.pool.has_primary_repositories()
+            poetry.pool.add_repository(PyPiRepository(), default, not default)
 
         return poetry
 
diff --git a/poetry/repositories/pool.py b/poetry/repositories/pool.py
--- a/poetry/repositories/pool.py
+++ b/poetry/repositories/pool.py
@@ -22,6 +22,7 @@ def __init__(
         self._lookup = {}  # type: Dict[str, int]
         self._repositories = []  # type: List[Repository]
         self._default = False
+        self._has_primary_repositories = False
         self._secondary_start_idx = None
 
         for repository in repositories:
@@ -38,6 +39,9 @@ def repositories(self):  # type: () -> List[Repository]
     def has_default(self):  # type: () -> bool
         return self._default
 
+    def has_primary_repositories(self):  # type: () -> bool
+        return self._has_primary_repositories
+
     def has_repository(self, name):  # type: (str) -> bool
         name = name.lower() if name is not None else None
 
@@ -81,6 +85,7 @@ def add_repository(
             self._repositories.append(repository)
             self._lookup[repository_name] = len(self._repositories) - 1
         else:
+            self._has_primary_repositories = True
             if self._secondary_start_idx is None:
                 self._repositories.append(repository)
                 self._lookup[repository_name] = len(self._repositories) - 1


Test patch:
diff --git a/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml b/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_multiple_secondary_sources/pyproject.toml
@@ -0,0 +1,24 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
+
+[[tool.poetry.source]]
+name = "bar"
+url = "https://bar.baz/simple/"
+secondary = true
diff --git a/tests/fixtures/with_non_default_multiple_sources/pyproject.toml b/tests/fixtures/with_non_default_multiple_sources/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_multiple_sources/pyproject.toml
@@ -0,0 +1,23 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
+
+[[tool.poetry.source]]
+name = "bar"
+url = "https://bar.baz/simple/"
diff --git a/tests/fixtures/with_non_default_secondary_source/pyproject.toml b/tests/fixtures/with_non_default_secondary_source/pyproject.toml
new file mode 100644
--- /dev/null
+++ b/tests/fixtures/with_non_default_secondary_source/pyproject.toml
@@ -0,0 +1,19 @@
+[tool.poetry]
+name = "my-package"
+version = "1.2.3"
+description = "Some description."
+authors = [
+    "Your Name <you@example.com>"
+]
+license = "MIT"
+
+# Requirements
+[tool.poetry.dependencies]
+python = "~2.7 || ^3.6"
+
+[tool.poetry.dev-dependencies]
+
+[[tool.poetry.source]]
+name = "foo"
+url = "https://foo.bar/simple/"
+secondary = true
diff --git a/tests/test_factory.py b/tests/test_factory.py
--- a/tests/test_factory.py
+++ b/tests/test_factory.py
@@ -166,6 +166,64 @@ def test_poetry_with_non_default_source():
     assert isinstance(poetry.pool.repositories[1], PyPiRepository)
 
 
+def test_poetry_with_non_default_secondary_source():
+    poetry = Factory().create_poetry(fixtures_dir / "with_non_default_secondary_source")
+
+    assert len(poetry.pool.repositories) == 2
+
+    assert poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+
+def test_poetry_with_non_default_multiple_secondary_sources():
+    poetry = Factory().create_poetry(
+        fixtures_dir / "with_non_default_multiple_secondary_sources"
+    )
+
+    assert len(poetry.pool.repositories) == 3
+
+    assert poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[2]
+    assert repository.name == "bar"
+    assert isinstance(repository, LegacyRepository)
+
+
+def test_poetry_with_non_default_multiple_sources():
+    poetry = Factory().create_poetry(fixtures_dir / "with_non_default_multiple_sources")
+
+    assert len(poetry.pool.repositories) == 3
+
+    assert not poetry.pool.has_default()
+
+    repository = poetry.pool.repositories[0]
+    assert repository.name == "bar"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[1]
+    assert repository.name == "foo"
+    assert isinstance(repository, LegacyRepository)
+
+    repository = poetry.pool.repositories[2]
+    assert repository.name == "PyPI"
+    assert isinstance(repository, PyPiRepository)
+
+
 def test_poetry_with_no_default_source():
     poetry = Factory().create_poetry(fixtures_dir / "sample_project")