Repository: pypa/pip

Issue:
--extra-index-url in requirements.txt
**Environment**

* pip version: 20.0.2 / 20.0.2
* Python version: 3.6.9 / 3.8.0
* OS: Linux Mint / MacOS
* keyring package: 21.2.0 / 21.2.0
* artifacts-keyring package: 0.2.9 / 0.2.9

**Description**
We are using an Azure DevOps artifact feed to provide our users with a 'mypackage' package. We're currently building applicationB that has this mypackage package as a dependency in the requirements.txt. 

To add the azure artifacts index to the install command, we've added `--extra-index-url https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/` at the top of the requirements.txt file and added the `mypackage` package to the requirements.txt list.

Running `pip install -r requirements.txt` gives me the following error:

```
Looking in indexes: https://pypi.org/simple, https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/
ERROR: Could not find a version that satisfies the requirement mypackage>=1.0.0 (from -r requirements.txt (line 2)) (from versions: none)
ERROR: No matching distribution found for mypackage>=1.0.0 (from -r requirements.txt (line 2))
```

Installing this package with an explicit --extra-index-url like so: `pip install -r requirements.txt --extra-index-url https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/` however, works perfectly and installs the packages like it should. 

Installing it also works when I add the extra index url to my global pip.conf settings.

**Expected behavior**
I expect all the dependencies (pypi and private azure artifacts feed) to be installed without issues using the `pip install -r requirements.txt` and without having to explicitly configure the index url in my pip configuration.

**How to Reproduce**

1. Set up a `requirements.txt` file with a link to a private pypi repository (azure artifacts?)
   For example:

   ```
   --extra-index-url https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/
   mypackage>=1.0.0
   requests
   ```

2. Then run `pip install -r requirements.txt`
3. An error occurs.

**Output**

```
Looking in indexes: https://pypi.org/simple, https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/
ERROR: Could not find a version that satisfies the requirement mypackage>=1.0.0 (from -r requirements.txt (line 2)) (from versions: none)
ERROR: No matching distribution found for mypackage>=1.0.0 (from -r requirements.txt (line 2))
```


Discussion:
+1
There's something inconsistent here. You state that `requirements.txt` contains

```
--extra-index-url https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/ mypackage==1.0.0 requests
```

But your output shows

```
ERROR: Could not find a version that satisfies the requirement mypackage>=1.6.1 (from -r requirements.txt (line 2))
```

Where did the requirement for 1.6.1 or greater come from?
@pfmoore You're right! I forgot to remove the version number, I fixed the error logs.
Hmm, OK. I'm not 100% sure why you needed to edit the version (generally, having as close a copy/paste of the actual situation where the error occurs is really helpful, because there can be key bits of information that might not be obvious at first glance) but I can't see why changing the version would be relevant here, so it's not a big deal.

Can you post the full log when running pip with `-vvv` added to the options, so we can see the complete logs and any information/debug output? Thanks.
## Without extra index url in command options:

```Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-6y85e8_9
Created temporary directory: /tmp/pip-req-tracker-b_tf3zns
Initialized build tracking at /tmp/pip-req-tracker-b_tf3zns
Created build tracker: /tmp/pip-req-tracker-b_tf3zns
Entered build tracker: /tmp/pip-req-tracker-b_tf3zns
Created temporary directory: /tmp/pip-install-fvd2fpmn
Looking in indexes: https://pypi.org/simple, https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/
2 location(s) to search for versions of mypackage:
* https://pypi.org/simple/mypackage/
* https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/mypackage/
Fetching project page and analyzing links: https://pypi.org/simple/mypackage/
Getting page https://pypi.org/simple/mypackage/
Found index url https://pypi.org/simple
Getting credentials from keyring for https://pypi.org/simple
Getting credentials from keyring for pypi.org
Looking up "https://pypi.org/simple/mypackage/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/mypackage/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/mypackage/: 404 Client Error: Not Found for url: https://pypi.org/simple/mypackage/ - skipping
Fetching project page and analyzing links: https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/mypackage/
Getting page https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/mypackage/
Getting credentials from keyring for pkgs.dev.azure.com
Looking up "https://pkgs.dev.azure.com/<org>/_packaging/mypackage/pypi/simple/mypackage/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pkgs.dev.azure.com:443
https://pkgs.dev.azure.com:443 "GET /<org>/_packaging/mypackage/pypi/simple/mypackage/ HTTP/1.1" 302 593
Status code 302 not in (200, 203, 300, 301)
Looking up "https://spsprodweu4.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F<org>%2F_packaging%2Fmypackage%2Fpypi%2Fsimple%2Fmypackage%2F&redirect=1&hid=<token>" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): spsprodweu4.vssps.visualstudio.com:443
https://spsprodweu4.vssps.visualstudio.com:443 "GET /_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F<org>%2F_packaging%2Fmypackage%2Fpypi%2Fsimple%2Fmypackage%2F&redirect=1&hid=<token> HTTP/1.1" 203 15654
Updating cache with response from "https://spsprodweu4.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F<org>%2F_packaging%2Fmypackage%2Fpypi%2Fsimple%2Fmypackage%2F&redirect=1&hid=<token>"
Response header has "no-store"
  Skipping link: not a file: https://spsprodweu4.vssps.visualstudio.com/go/profile (from https://spsprodweu4.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F<org>%2F_packaging%2Fmypackage%2Fpypi%2Fsimple%2Fmypackage%2F&redirect=1&hid=<token>#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ud2luZG93cy5uZXQiXX01)
  Skipping link: not a file: https://spsprodweu4.vssps.visualstudio.com/_signout (from https://spsprodweu4.vssps.visualstudio.com/_signin?realm=pkgs.dev.azure.com&reply_to=https%3A%2F%2Fpkgs.dev.azure.com%2F<org>%2F_packaging%2Fmypackage%2Fpypi%2Fsimple%2Fmypackage%2F&redirect=1&hid=68f47bca-fbc6-48e0-81db-6958e3f16b5b&context=eyJodCI6MiwiaGlkIjoiZDZhOTRkYTYtOTVkYS00NjViLTg2Y2UtINGCNDkzNWIzNTQ1M2RlIiwicXMiOnt9LCJyciI6IiIsInZoIjoiIiwiY3YiOiIiLCJjcyI6IiJ90#ctx=eyJTaWduSW5Db29raWVEb21haW5zIjpbImh0dHBzOi8vbG9naW4ud2luZG93cy5uZXQiXX01)
Given no hashes to check 0 links for project 'mypackage': discarding no candidates
ERROR: Could not find a version that satisfies the requirement mypackage>=1.6.1 (from -r requirements.txt (line 2)) (from versions: none)
Cleaning up...
Removed build tracker: '/tmp/pip-req-tracker-b_tf3zns'
ERROR: No matching distribution found for mypackage>=1.6.1 (from -r requirements.txt (line 2))
Exception information:
Traceback (most recent call last):
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
    status = self.run(options, args)
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 331, in run
    resolver.resolve(requirement_set)
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/legacy_resolve.py", line 281, in _get_abstract_dist_for
    req.populate_link(self.finder, upgrade_allowed, require_hashes)
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/req/req_install.py", line 249, in populate_link
    self.link = finder.find_requirement(self, upgrade)
  File "/home/maarten/dev/mypackage/venv/lib/python3.6/site-packages/pip/_internal/index/package_finder.py", line 927, in find_requirement
    'No matching distribution found for %s' % req
pip._internal.exceptions.DistributionNotFound: No matching distribution found for mypackage>=1.6.1 (from -r requirements.txt (line 2))
```

When appending the --extra-index-url to the pip install command it asks me to login to azure and afterwards it installs the package like normal. In these logs I kept the version number in.
It looks like it might be getting a signin page and treating it as an index. That's likely your problem, for some reason with `--extra-index-url` in the requirements file, the keyring stuff isn't kicking in properly.

I suspect that the `PipSession` being built is only being populated with the index URLs from the command line, not the ones from the requirements file, and the keyring handling is hooked into that part of the process. But I don't know enough about the keyring support in pip to be sure.

@zooba what's your view here, as you implemented the keyring support?
Apologies for hijacking but seems like the same happens with `-f` inside `requirements.txt`.

    # requirements.txt
    torch==1.5.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

    pip install -r requirements.txt

    Exception information:
    Traceback (most recent call last):
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\cli\base_command.py", line 188, in _main
        status = self.run(options, args)
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\cli\req_command.py", line 185, in wrapper
        return func(self, options, args)
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\commands\install.py", line 333, in run
        reqs, check_supported_wheels=not options.target_dir
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 179, in resolve
    discovered_reqs.extend(self._resolve_one(requirement_set, req))
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 362, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 313, in _get_abstract_dist_for
    self._populate_link(req)
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\resolution\legacy\resolver.py", line 279, in _populate_link
    req.link = self.finder.find_requirement(req, upgrade)
      File "D:\Tools\conda\envs\u8-test\lib\site-packages\pip\_internal\index\package_finder.py", line 930, in find_requirement
    req)
    pip._internal.exceptions.DistributionNotFound: No matching distribution found for torch==1.5.0+cpu
    
While this works:

    pip install -r requirements.txt -f https://download.pytorch.org/whl/torch_stable.html

Is the syntax perhaps different?

Resolving patch:
diff --git a/src/pip/_internal/network/session.py b/src/pip/_internal/network/session.py
--- a/src/pip/_internal/network/session.py
+++ b/src/pip/_internal/network/session.py
@@ -305,6 +305,14 @@ def __init__(self, *args, **kwargs):
         for host in trusted_hosts:
             self.add_trusted_host(host, suppress_logging=True)
 
+    def update_index_urls(self, new_index_urls):
+        # type: (List[str]) -> None
+        """
+        :param new_index_urls: New index urls to update the authentication
+            handler with.
+        """
+        self.auth.index_urls = new_index_urls
+
     def add_trusted_host(self, host, source=None, suppress_logging=False):
         # type: (str, Optional[str], bool) -> None
         """
diff --git a/src/pip/_internal/req/req_file.py b/src/pip/_internal/req/req_file.py
--- a/src/pip/_internal/req/req_file.py
+++ b/src/pip/_internal/req/req_file.py
@@ -256,6 +256,10 @@ def handle_option_line(
                 value = relative_to_reqs_file
             find_links.append(value)
 
+        if session:
+            # We need to update the auth urls in session
+            session.update_index_urls(index_urls)
+
         search_scope = SearchScope(
             find_links=find_links,
             index_urls=index_urls,


Test patch:
diff --git a/tests/unit/test_req_file.py b/tests/unit/test_req_file.py
--- a/tests/unit/test_req_file.py
+++ b/tests/unit/test_req_file.py
@@ -341,17 +341,22 @@ def test_set_finder_no_index(self, line_processor, finder):
         line_processor("--no-index", "file", 1, finder=finder)
         assert finder.index_urls == []
 
-    def test_set_finder_index_url(self, line_processor, finder):
-        line_processor("--index-url=url", "file", 1, finder=finder)
+    def test_set_finder_index_url(self, line_processor, finder, session):
+        line_processor(
+            "--index-url=url", "file", 1, finder=finder, session=session)
         assert finder.index_urls == ['url']
+        assert session.auth.index_urls == ['url']
 
     def test_set_finder_find_links(self, line_processor, finder):
         line_processor("--find-links=url", "file", 1, finder=finder)
         assert finder.find_links == ['url']
 
-    def test_set_finder_extra_index_urls(self, line_processor, finder):
-        line_processor("--extra-index-url=url", "file", 1, finder=finder)
+    def test_set_finder_extra_index_urls(
+            self, line_processor, finder, session):
+        line_processor(
+            "--extra-index-url=url", "file", 1, finder=finder, session=session)
         assert finder.index_urls == ['url']
+        assert session.auth.index_urls == ['url']
 
     def test_set_finder_trusted_host(
         self, line_processor, caplog, session, finder