Repository: pypa/pip

Issue:
Minor: Undefined behavior leads to confusing error with --index-url arg.
**Environment**

* pip version: 19.2.3 
* Python version: 3.8
* OS: Arch Linux

I do not believe this is limited to arch, but it was discovered as such here: https://github.com/SalomonSmeke/rere/issues/5

_note: as pointed out by @uranusjr [below](https://github.com/pypa/pip/issues/7430#issuecomment-561975264) this is partially fixed in 19.3_

**Description**

A very minor problem, borderline not a bug but simply less than ideal behavior. When using a --index-url with pip install, passing something other than a url causes a confusing error.

This can manifest like in the issue above:
`pip install --index-url --user https://test.pypi.org/simple/ reretrieve`
when trying to:  
`pip install --user --index-url https://test.pypi.org/simple/ reretrieve`

**Expected behavior**

An error about how the index url is invalid, not found, or otherwise messed up.

**How to Reproduce**

1. pip install --index-url --user https://test.pypi.org/simple/ reretrieve

**Output**

```console
$ pip install --index-url --user https://test.pypi.org/simple/ reretrieve
Looking in indexes: --user
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-w9t0tixn/simple (downloaded from /tmp/pip-req-build-s5ub6ixa, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-s5ub6ixa.
```

_note: as pointed out by @uranusjr [below](https://github.com/pypa/pip/issues/7430#issuecomment-561975264) in 19.3 the output will look a little different._

**Edit**

I just wanted to clarify that this happens with any bad text input/invalid url, not just `--user` and such.

```console
$ pip install --index-url foo  https://test.pypi.org/simple/ reretrieve
Looking in indexes: foo
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-160ive4r/simple (downloaded from /tmp/pip-req-build-8dw7kt5w, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-8dw7kt5w
```

I dont blame you if you decide to `wontfix` this one, but wanted to make sure it was an intentional decision. thank you in advance for taking a look!


Discussion:
pip 19.3 has a warning like this:

```
WARNING: Url 'foo/pip/' is ignored. It is either a non-existing path or lacks a specific scheme.
```

that would probably help debug the problem. I kind of feel it can do a better job though; maybe detect the malformed index URL before it gets concatenated to the package name.

The *cannot determine archive format* error is much more complicated, unfortunately. pip allows specifying a package by URL, e.g.

```
pip install https://github.com/pypa/pip/archive/master.zip
```

and pip has no way to know whether you actually want to install `https://test.pypi.org/simple/` as a package. So that part should probably be *wontfix*.
@uranusjr of course the next version from mine got the warning! That would have definitely saved me some time in debugging, but I agree that ideally the message is clearer.

`cannot determine archive format` is unimportant here. if something like:

```console
$ pip install --index-url --user https://test.pypi.org/simple/ reretrieve
ERROR: index-url --user is invalid.
Collecting https://test.pypi.org/simple/
  Using cached https://test.pypi.org/simple/
  ERROR: Cannot unpack file /tmp/pip-unpack-w9t0tixn/simple (downloaded from /tmp/pip-req-build-s5ub6ixa, content-type: text/html; charset=UTF-8); cannot detect archive format
ERROR: Cannot determine archive format of /tmp/pip-req-build-s5ub6ixa.
```

happens, the true source of the problem would be immediately clear to me, or at worst just a lot less difficult to track down.

In fact, I would argue `cannot determine archive format` is accurate and should remain! Assuming pip would not rather just `exit` after `ERROR: index-url --user is invalid.`.
I'm +1 on mentioning `--index-url` in the warning message, assuming it doesn't make a mess in the code.

People can have `--index-url` and `--extra-index-url` in `requirements.txt` which may point to directories that don't always exist, so I would be wary of turning this into a hard failure.
Hi @chrahunt , @uranusjr 

Can I work on this PR? Also I was thinking about what would be the best place to check for such an issue.

I was thinking that we can insert an error in the below function, when we are checking the index urls. Or maybe there is a better place to put this error perhaps?

https://github.com/pypa/pip/blob/6086f71cde81454fda588c20868a241561809836/src/pip/_internal/models/search_scope.py#L77-L84
I’m not entirely sure, to be honest. I think part of the reason this is marked as *awaiting PR* is because it’s difficult to gauge the situation without diving into the implementation, and once you do that you’re in the best position to produce a PR. So I’d say go ahead writing some code first and we can see what we can work out.

Resolving patch:
diff --git a/src/pip/_internal/models/search_scope.py b/src/pip/_internal/models/search_scope.py
--- a/src/pip/_internal/models/search_scope.py
+++ b/src/pip/_internal/models/search_scope.py
@@ -77,11 +77,30 @@ def __init__(
     def get_formatted_locations(self):
         # type: () -> str
         lines = []
+        redacted_index_urls = []
         if self.index_urls and self.index_urls != [PyPI.simple_url]:
-            lines.append(
-                'Looking in indexes: {}'.format(', '.join(
-                    redact_auth_from_url(url) for url in self.index_urls))
-            )
+            for url in self.index_urls:
+
+                redacted_index_url = redact_auth_from_url(url)
+
+                # Parse the URL
+                purl = urllib_parse.urlsplit(redacted_index_url)
+
+                # URL is generally invalid if scheme and netloc is missing
+                # there are issues with Python and URL parsing, so this test
+                # is a bit crude. See bpo-20271, bpo-23505. Python doesn't
+                # always parse invalid URLs correctly - it should raise
+                # exceptions for malformed URLs
+                if not purl.scheme and not purl.netloc:
+                    logger.warning(
+                        'The index url "{}" seems invalid, '
+                        'please provide a scheme.'.format(redacted_index_url))
+
+                redacted_index_urls.append(redacted_index_url)
+
+            lines.append('Looking in indexes: {}'.format(
+                ', '.join(redacted_index_urls)))
+
         if self.find_links:
             lines.append(
                 'Looking in links: {}'.format(', '.join(


Test patch:
diff --git a/tests/functional/test_install.py b/tests/functional/test_install.py
--- a/tests/functional/test_install.py
+++ b/tests/functional/test_install.py
@@ -1769,6 +1769,31 @@ def test_ignore_yanked_file(script, data):
     assert 'Successfully installed simple-2.0\n' in result.stdout, str(result)
 
 
+def test_invalid_index_url_argument(script, shared_data):
+    """
+    Test the behaviour of an invalid --index-url argument
+    """
+
+    result = script.pip('install', '--index-url', '--user',
+                        shared_data.find_links3, "Dinner",
+                        expect_error=True)
+
+    assert 'WARNING: The index url "--user" seems invalid, ' \
+           'please provide a scheme.' in result.stderr, str(result)
+
+
+def test_valid_index_url_argument(script, shared_data):
+    """
+    Test the behaviour of an valid --index-url argument
+    """
+
+    result = script.pip('install', '--index-url',
+                        shared_data.find_links3,
+                        "Dinner")
+
+    assert 'Successfully installed Dinner' in result.stdout, str(result)
+
+
 def test_install_yanked_file_and_print_warning(script, data):
     """
     Test install a "yanked" file and print a warning.