Repository: pypa/pipenv

Issue:
Another possible `install_search_all_sources = true` regression
There is this associated issue: https://github.com/pypa/pipenv/issues/5743
It was closed, but I think this is a little different so figured I'd raise a new issue for it...

### Issue description

In pipenv version 2023.6.18 (and onwards), attempting to run `pipenv sync --dev` (`--dev` flag likely unimportant) raises an error only when the following is in the pipfile:
```
[pipenv]
install_search_all_sources = true
```
I have a number of dependencies in private repos and all have their repos explictly specified, as well as any sub-dependencies which are also in our private repo. The issue is NOT with locking. This works perfectly fine. The issue is only when calling `sync`.

### Expected result

I'd not expect this flag to affect syncing, least of all cause it to fail.

### Actual result

```
Installing initially failed dependencies...
Looking in indexes: https://pypi.org/simple, <private_index_url_1>, <private_index_url_2>, <private_index_url_1>, <private_index_url_2>
...
[pipenv.exceptions.InstallError]: ERROR: Could not find a version that satisfies the requirement <private_repo>==0.1.dev19+g92d125f (from versions: none)
[pipenv.exceptions.InstallError]: ERROR: No matching distribution found for <private_repo>==0.1.dev19+g92d125f
ERROR: Couldn't install package: [Requirement(_name='alembic', vcs=None, req=NamedRequirement(name='alembic', version='==1.11.1', req=Requirement.parse('alembic==1.11.1'), extras=[], editable=False, _parsed_line=<Line (editable=False, name=alembic, path=None, uri=None, extras=(), markers=None, vcs=None, specifier===1.11.1, ...
```

(yes, the private repo urls are listed twice like that... Looks a bit weird...)

### Steps to replicate

See description.
Locking the file works fine in all versions. All the dependencies are resolved correctly and the lock file looks correct (private dependencies have the correct index urls in the lock file).
Running `pipenv sync --dev` in version 2023.6.18 onward fails when `install_search_all_sources = true` is present, but passes when it isn't, whereas 2023.6.12 has no issues.

This isn't the biggest issue as it's easy to remove and we don't need it since we have all the dependencies specified, but it did take a while to figure out why it was happening, and I'm mostly confused as to why this is even happening...


Discussion:
Admittedly I was confused too when I had to update the integration tests to not use the `install_search_all_sources = true` option with https://github.com/pypa/pipenv/pull/5737

I believe what is happening is that in the `install_search_all_sources = true` code path, all indexes are available for pip to search for the install, but with my patch related to https://github.com/pypa/pip/issues/11784 causes the search scope to be reduced to just the first index -- it is because the index lookups aren't passed during install (because its not a real pip feature yet) it gets restricted from finding what its looking for because its simply not in the lookup dictionary.   I evaluated trying to add it, but it the proper way would be supporting it in pip to pass the `--index` per requirement line being specified for install -- this is basically what the batch install logic in pipenv does when `install_search_all_sources = false` which is the default behavior.
I would be curious if your result @monkeyman192 is any different with the newest pipenv release from today.
@matteius I just tried with `2023.8.20` and still the same issue.

Resolving patch:
diff --git a/pipenv/patched/pip/_internal/models/search_scope.py b/pipenv/patched/pip/_internal/models/search_scope.py
--- a/pipenv/patched/pip/_internal/models/search_scope.py
+++ b/pipenv/patched/pip/_internal/models/search_scope.py
@@ -20,7 +20,7 @@ class SearchScope:
     Encapsulates the locations that pip is configured to search.
     """
 
-    __slots__ = ["find_links", "index_urls", "no_index", "index_lookup"]
+    __slots__ = ["find_links", "index_urls", "no_index", "index_lookup", "index_restricted"]
 
     @classmethod
     def create(
@@ -29,6 +29,7 @@ def create(
         index_urls: List[str],
         no_index: bool,
         index_lookup: Optional[Dict[str, List[str]]] = None,
+        index_restricted: bool = False,
     ) -> "SearchScope":
         """
         Create a SearchScope object after normalizing the `find_links`.
@@ -64,6 +65,7 @@ def create(
             index_urls=index_urls,
             no_index=no_index,
             index_lookup=index_lookup,
+            index_restricted=index_restricted,
         )
 
     def __init__(
@@ -72,11 +74,13 @@ def __init__(
         index_urls: List[str],
         no_index: bool,
         index_lookup: Optional[Dict[str, List[str]]] = None,
+        index_restricted: bool = False,
     ) -> None:
         self.find_links = find_links
         self.index_urls = index_urls
         self.no_index = no_index
         self.index_lookup = index_lookup if index_lookup else {}
+        self.index_restricted = index_restricted
 
     def get_formatted_locations(self) -> str:
         lines = []
@@ -136,6 +140,6 @@ def mkurl_pypi_url(url: str) -> str:
         index_urls = self.index_urls
         if project_name in self.index_lookup:
             index_urls = [self.index_lookup[project_name]]
-        elif self.index_urls:
+        elif self.index_restricted and self.index_urls:
             index_urls = [self.index_urls[0]]
         return [mkurl_pypi_url(url) for url in index_urls]
diff --git a/pipenv/utils/resolver.py b/pipenv/utils/resolver.py
--- a/pipenv/utils/resolver.py
+++ b/pipenv/utils/resolver.py
@@ -245,7 +245,6 @@ def create(
                 resolver.skipped[package_name] = dep
         resolver.initial_constraints = constraints
         resolver.index_lookup = index_lookup
-        resolver.finder.index_lookup = index_lookup
         resolver.markers_lookup = markers_lookup
         return resolver
 
@@ -340,6 +339,7 @@ def finder(self):
         finder = self.package_finder
         index_lookup = self.prepare_index_lookup()
         finder._link_collector.index_lookup = index_lookup
+        finder._link_collector.search_scope.index_restricted = True
         finder._link_collector.search_scope.index_lookup = index_lookup
         return finder
 


Test patch:
diff --git a/tests/integration/test_install_uri.py b/tests/integration/test_install_uri.py
--- a/tests/integration/test_install_uri.py
+++ b/tests/integration/test_install_uri.py
@@ -139,8 +139,8 @@ def test_install_named_index_alias(pipenv_instance_private_pypi):
 @pytest.mark.install
 @pytest.mark.needs_internet
 @pytest.mark.skipif(sys.version_info >= (3, 12), reason="Package does not work with Python 3.12")
-def test_install_specifying_index_url(pipenv_instance_pypi):
-    with pipenv_instance_pypi() as p:
+def test_install_specifying_index_url(pipenv_instance_private_pypi):
+    with pipenv_instance_private_pypi() as p:
         with open(p.pipfile_path, "w") as f:
             contents = """
 [[source]]
@@ -153,6 +153,8 @@ def test_install_specifying_index_url(pipenv_instance_pypi):
 
 [dev-packages]
 
+[pipenv]
+install_search_all_sources = true
             """.strip()
             f.write(contents)
         c = p.pipenv("install pipenv-test-private-package --index https://test.pypi.org/simple")
diff --git a/tests/integration/test_lock.py b/tests/integration/test_lock.py
--- a/tests/integration/test_lock.py
+++ b/tests/integration/test_lock.py
@@ -647,3 +647,31 @@ def test_pinned_pipfile_no_null_markers_when_extras(pipenv_instance_pypi):
         assert "dataclasses-json" in p.pipfile["packages"]
         assert "dataclasses-json" in p.lockfile["default"]
         assert p.lockfile["default"]["dataclasses-json"].get("markers", "") is not None
+
+@pytest.mark.index
+@pytest.mark.install  # private indexes need to be uncached for resolution
+@pytest.mark.skip_lock
+@pytest.mark.needs_internet
+def test_private_index_skip_lock(pipenv_instance_private_pypi):
+    with pipenv_instance_private_pypi() as p:
+        with open(p.pipfile_path, 'w') as f:
+            contents = """
+[[source]]
+url = "https://pypi.org/simple"
+verify_ssl = true
+name = "pypi"
+
+[[source]]
+url = "https://test.pypi.org/simple"
+verify_ssl = true
+name = "testpypi"
+
+[packages]
+pipenv-test-private-package = {version = "*", index = "testpypi"}
+
+[pipenv]
+install_search_all_sources = true
+            """.strip()
+            f.write(contents)
+        c = p.pipenv('install --skip-lock')
+        assert c.returncode == 0