Repository: pypa/pipenv

Issue:
"Could not find a version that satisfies the requirement" for package in private repository starting from pipenv==2022.8.31 
### Issue description

With a Pipfile like this...
```
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://pypi.ourinternalrepo.com/simple"
verify_ssl = true
name = "our-pypi"

[packages]
..etc..
ourinternalpackage = {version = "==0.5.9", index = "our-pypi"}
```
that generates a Pipfile.lock like this...
```
{
    "_meta": {
        "hash": {
            "sha256": "something"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.8"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            },
            {
                "name": "our-pypi",
                "url": "https://pypi.ourinternalrepo.com/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "ourinternalpackage ": {
            "hashes": [
                "sha256:something"
            ],
            "index": "our-pypi",
            "version": "==0.1.1"
        },
        ...etc
```

but when we execute `pipenv sync --verbose`, it generates a pip install command like so

```
/usr/local/lib/python3.8/site-packages/pipenv/patched/pip/__pip-runner__.py install -i https://pypi.org/simple --no-input --upgrade --no-deps --exists-action=i -r /tmp/pipenv-etc-requirements/pipenv-etc-hashed-reqs.txt
```

(note "-i https://pypi.org/simple")

and as might be expected, this fails with `6.552 ERROR: Could not find a version that satisfies the requirement ourinternalpackage==0.1.1 (from versions: none)`. This appears to have started happening with 2022.8.31--when we `pip install -U pipenv==2022.8.30` and run `pipenv sync` from there, the package resolves successfully.

### Expected result

I would expect the package install to generate with a -i pointed at the `our-pypi` source rather than the default.

### Actual result

```
6.552 ERROR: Could not find a version that satisfies the requirement ourinternalpackage==0.1.1 (from versions: none)
```

There are also bunches of these, that may or may not be irrelevant. I can't provide the full --verbose output because cleansing it of work-internal information would be too much hassle, but I have it around if anyone wants specific snippets.

```
An error occurred while installing alembic==1.8.0; python_version >= '3.7' --hash=sha256:a2d4d90da70b30e70352cd9455e35873a255a31402a438fe24815758d7a0e5e1 --hash=sha256:b5ae4bbfc7d1302ed413989d39474d102e7cfa158f6d5969d2497955ffe85a30! Will try again.
```
### Steps to replicate

See above

-------------------------------------------------------------------------------

Please run `$ pipenv --support`, and paste the results here. Don't put backticks (`` ` ``) around it! The output already contains Markdown formatting.
<details><summary>$ pipenv --support</summary>

Pipenv version: `'2022.9.8'`

Pipenv location: `'/usr/local/lib/python3.8/site-packages/pipenv'`

Python location: `'/usr/local/bin/python'`

OS Name: `'posix'`

User pip version: `'22.2.2'`

user Python installations found:

  - `3.8.14`: `/usr/local/bin/python`
  - `3.8.14`: `/usr/local/bin/python3`
  - `3.8.14`: `/usr/local/bin/python3.8`
  - `3.7.3`: `/usr/bin/python3`
  - `3.7.3`: `/usr/bin/python3.7`
  - `3.7.3`: `/usr/bin/python3.7m`
  - `2.7.16`: `/usr/bin/python`
  - `2.7.16`: `/usr/bin/python2.7`
  - `2.7.16`: `/usr/bin/python2`

PEP 508 Information:

```
{'implementation_name': 'cpython',
 'implementation_version': '3.8.14',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.4.0-1051-gcp',
 'platform_system': 'Linux',
 'platform_version': '#55~18.04.1-Ubuntu SMP Sun Aug 1 20:38:04 UTC 2021',
 'python_full_version': '3.8.14',
 'python_version': '3.8',
 'sys_platform': 'linux'}
```

System environment variables:

  - `PIPENV_VENV_IN_PROJECT`
  - `HOSTNAME`
  - `PYTHON_VERSION`
  - `PWD`
  - `PYTHON_SETUPTOOLS_VERSION`
  - `PIPENV_CACHE_DIR`
  - `HOME`
  - `LANG`
  - `LS_COLORS`
  - `GPG_KEY`
  - `PYTHONPATH`
  - `TERM`
  - `SHLVL`
  - `PYTHON_PIP_VERSION`
  - `PYTHON_GET_PIP_SHA256`
  - `PYTHON_GET_PIP_URL`
  - `PATH`
  - `_`
  - `PIP_DISABLE_PIP_VERSION_CHECK`
  - `PIP_PYTHON_PATH`
  - `PYTHONDONTWRITEBYTECODE`
  - `PYTHONFINDER_IGNORE_UNSUPPORTED`

Pipenv–specific environment variables:

 - `PIPENV_VENV_IN_PROJECT`: `1`
 - `PIPENV_CACHE_DIR`: `/tmp`

Debug–specific environment variables:

  - `PATH`: `/usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin`
  - `LANG`: `C.UTF-8`
  - `PWD`: `/code`


---------------------------

Contents of `Pipfile` ('/code/Pipfile'):

```toml
[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

[dev-packages]

[requires]
python_version = "3.8"

```


Contents of `Pipfile.lock` ('/code/Pipfile.lock'):

(see above for selected snippets, full output not included for reasons stated above)



Discussion:
perusing the output above, I'm seeing that the contents of the Pipfile there are completely different than they should be (even though Pipfile.lock is correct; it should contain the same source entry mentioned above), I'll investigate more on that front
Ok, I see that the root issue seems to be that we were only copying `Pipfile.lock` into the container, rather than `Pipfile`. When I add a `COPY` statement for the `Pipfile` as well, `pipenv sync` works as expected. This changes the bug report: Why does `pipenv sync` suddenly need a `Pipfile` to work correctly, rather than being able to correctly resolve all dependencies with the provided `Pipfile.lock`?
@aawilson I am not sure that this is a new behavior, so there may be something else going on, but you found that having the Pipfile allows it to work?   It is an interesting observation, apparently `get_source_list` gets its sources from the `Pipfile`, even though its a method that appears to only be used in the install phase.   This is not a new behavior though 😕  We should improve that though to pull the sources from just the lock file. 
It's pretty interesting, I definitely get the "broken" behavior in 2022.8.31 and not in 2022.8.30 (since it's a container it's easy to remake the intermediate image up to failure and try different versions). Something seems to have changed between those two versions to get the specified behavior.
@aawilson Just to clarify -- does it work to install if you include the `Pipfile` as well?
Inside `pip_install_deps`, we call `get_source_list` with `index=None` and `extra_indexes=None`
https://github.com/pypa/pipenv/blob/9572c319fd3eed0eece9bb1ee6a8ace107e4f8fc/pipenv/core.py#L1638-L1643

Therefore `get_source_list` will get source from `Pipfile` instead of `Pipfile.lock` (line 91)
https://github.com/pypa/pipenv/blob/9572c319fd3eed0eece9bb1ee6a8ace107e4f8fc/pipenv/utils/indexes.py#L68-L98

To address this issue, there's a fix that is throwing all the indexes written inside Pipfile.lock into `index` and `extra_indexes` from the call of `get_source_list`.
But in that case, all `index` and `extra_indexes` will be passed in `pip_command`, therefore we could not declare index for specific package anymore. 

This could lead to an issue like: "I want to install A from index X, but A is installed from index Y. I don't want that because A from Y is broken."
> To address this issue, there's a fix that is throwing all the indexes written inside Pipfile.lock into `index` and `extra_indexes` from the call of `get_source_list`. 

Yes, this is what I am thinking `get_source_list` should be getting its sources from the lock file, not the `Pipfile`.

> But in that case, all `index` and `extra_indexes` will be passed in `pip_command`, therefore we could not declare index for specific package anymore.

This is not quite accurate -- the resolver was patched for the specific indexes, but once packages resolve they are hashed.


> This could lead to an issue like: "I want to install A from index X, but A is installed from index Y. I don't want that because A from Y is broken."

Because the normal index specified packages are hashed, the install phase is less sensitive -- If you are installing package A with hash Z then it has to be the same package whether you get it from X or Y -- that is how it works today with batch install as long as you have your `Pipfile` in the project with all of its sources, so changing where we get the sources from shouldn't matter.  

In fact the default behavior is to search all sources.   For regular `pip_install` we check `if not search_all_sources and requirement.index in source_names:` and restrict to just that index, but we cannot do this for batch install because its more than one item in the batch.   If there is a real use case for it, we may have to add additional batches per index if the setting `if not search_all_sources` but I think that could be addressed outside of an effort to make sure that `get_source_list` is built from `Pipfile.lock`.

> @aawilson Just to clarify -- does it work to install if you include the `Pipfile` as well?

Yeah, it's up there but to confirm explicitly, adding Pipfile to the `COPY` does cause `sync` to succeed.

Resolving patch:
diff --git a/pipenv/utils/indexes.py b/pipenv/utils/indexes.py
--- a/pipenv/utils/indexes.py
+++ b/pipenv/utils/indexes.py
@@ -84,12 +84,12 @@ def get_source_list(
             if not sources or extra_src["url"] != sources[0]["url"]:
                 sources.append(extra_src)
 
-        for source in project.pipfile_sources:
+        for source in project.sources:
             if not sources or source["url"] != sources[0]["url"]:
                 sources.append(source)
 
     if not sources:
-        sources = project.pipfile_sources[:]
+        sources = project.sources[:]
     if pypi_mirror:
         sources = [
             create_mirror_source(pypi_mirror) if is_pypi_url(source["url"]) else source


Test patch: