Repository: pypa/pip
Pull request: Fix redact url in --help
Closes #9191 
Closes #9196

Happened as we pass a list from --extra-index-url values, and we only handled regular strings
Added a test to address this

@pradyunsg @sbidoul 

Problem statement:
pip install --help fails
<!--
If you're reporting an issue for `--use-feature=2020-resolver`, use the "Dependency resolver failures / errors" template instead.
-->

**Environment**

* pip version: 20.3
* Python version: 3.7.9 & 2.7.16
* OS: macOS Catalina 10.15.7

<!-- Feel free to add more information about your environment here -->

**Description**
`pip install --help` fails to parse the cli args, resulting in it raising with an `AttributeError`, for both py2.7 and py3.7
<!-- A clear and concise description of what the bug is. -->

**Expected behavior**
normal help output for the `install` command, as in pip 20.2.4:
```
$ pip install --help

Usage:
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:

  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.

  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This option can be used multiple times.
{...}
```
<!-- A clear and concise description of what you expected to happen. -->

**How to Reproduce**
<!-- Describe the steps to reproduce this bug. -->

1. set up and activate fresh virtual env 
`$ python -m virtualenv ve && . ve/bin/activate`
2. install old pip 
`$ pip install pip==20.2.4`
3. get expected help output
`$ pip install --help`
4. install new pip
`$ pip install pip==20.3`
5. try again, should fail
`$ pip install --help`

**Output**

for python 3.7.9, `pip install --help` fails with the following error:
```
$ pip install --help
Traceback (most recent call last):
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/bin/pip", line 8, in <module>
    sys.exit(main())
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/main.py", line 75, in main
    return command.main(cmd_args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 117, in main
    return self._main(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 131, in _main
    options, args = self.parse_args(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 111, in parse_args
    return self.parser.parse_args(args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1387, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1427, in _process_args
    self._process_long_opt(rargs, values)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1501, in _process_long_opt
    option.process(opt, value, values, self)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 785, in process
    self.action, self.dest, opt, value, values, parser)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 807, in take_action
    parser.print_help()
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1647, in print_help
    file.write(self.format_help())
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1635, in format_help
    result.append(self.format_option_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1618, in format_option_help
    result.append(group.format_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1101, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1072, in format_help
    result.append(self.format_option_help(formatter))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 1061, in format_option_help
    result.append(formatter.format_option(option))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/optparse.py", line 322, in format_option
    help_text = self.expand_default(option)
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/cli/parser.py", line 123, in expand_default
    default_value, redact_auth_from_url(default_value))
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 826, in redact_auth_from_url
    return _transform_url(url, _redact_netloc)[0]
  File "/Users/rblanchard/repos/invitae-evidence-service/ve/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 786, in _transform_url
    purl = urllib_parse.urlsplit(url)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 418, in urlsplit
    url, scheme, _coerce_result = _coerce_args(url, scheme)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 123, in _coerce_args
    return _decode_args(args) + (_encode_result,)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in _decode_args
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/urllib/parse.py", line 107, in <genexpr>
    return tuple(x.decode(encoding, errors) if x else '' for x in args)
AttributeError: 'list' object has no attribute 'decode'
```

using python 2.7.16 produces a similar, but slightly different error:
```
$ pip install --help
Traceback (most recent call last):
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/bin/pip", line 8, in <module>
    sys.exit(main())
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/main.py", line 75, in main
    return command.main(cmd_args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 117, in main
    return self._main(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 131, in _main
    options, args = self.parse_args(args)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 111, in parse_args
    return self.parser.parse_args(args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1400, in parse_args
    stop = self._process_args(largs, rargs, values)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1440, in _process_args
    self._process_long_opt(rargs, values)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1515, in _process_long_opt
    option.process(opt, value, values, self)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 789, in process
    self.action, self.dest, opt, value, values, parser)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 811, in take_action
    parser.print_help()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1670, in print_help
    file.write(self.format_help().encode(encoding, "replace"))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1650, in format_help
    result.append(self.format_option_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1633, in format_option_help
    result.append(group.format_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1114, in format_help
    result += OptionContainer.format_help(self, formatter)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1085, in format_help
    result.append(self.format_option_help(formatter))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 1074, in format_option_help
    result.append(formatter.format_option(option))
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/optparse.py", line 316, in format_option
    help_text = self.expand_default(option)
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/cli/parser.py", line 123, in expand_default
    default_value, redact_auth_from_url(default_value))
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 826, in redact_auth_from_url
    return _transform_url(url, _redact_netloc)[0]
  File "/Users/rblanchard/repos/invitae-evidence-service/v2/lib/python2.7/site-packages/pip/_internal/utils/misc.py", line 786, in _transform_url
    purl = urllib_parse.urlsplit(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urlparse.py", line 176, in urlsplit
    cached = _parse_cache.get(key, None)
TypeError: unhashable type: 'list'
```

Hints from issue discussion:
HUH. This is definitely worth fixing. ;)

Thanks for the bug report @Rosswell! Would it be possible to provide the output of `pip config list`?
@pradyunsg sure thing:
```
global.extra-index-url='https://pypi.python.org/simple/'
global.index-url='https://nexus.it.locusdev.net/repository/invitae-pypi/simple/'
global.no-cache-dir='true'
global.retries='1'
global.timeout='2'
global.trusted-host='nexus.it.locusdev.net'
```
I figured it would prolly be a good first issue, but didn't want to presume without looking into the cli parsing
#9163 touched that area in 20.3
I'd start by reverting #9163, and we can revisit it after. :)

Commit message: Handle case of list default values in UpdatingDefaultsHelpFormatter

Happens because we pass a list from --extra-index-url

Patch to review:
diff --git a/news/9191.bugfix.rst b/news/9191.bugfix.rst
new file mode 100644
index 00000000000..436f2f2dd29
--- /dev/null
+++ b/news/9191.bugfix.rst
@@ -0,0 +1,2 @@
+Handle case of list default values in UpdatingDefaultsHelpFormatter
+Happens because we pass a list from --extra-index-url
diff --git a/src/pip/_internal/cli/parser.py b/src/pip/_internal/cli/parser.py
index ea3b383e2f7..7170bfd3841 100644
--- a/src/pip/_internal/cli/parser.py
+++ b/src/pip/_internal/cli/parser.py
@@ -112,15 +112,23 @@ class UpdatingDefaultsHelpFormatter(PrettyHelpFormatter):
     """
 
     def expand_default(self, option):
-        default_value = None
+        default_values = None
         if self.parser is not None:
             self.parser._update_defaults(self.parser.defaults)
-            default_value = self.parser.defaults.get(option.dest)
+            default_values = self.parser.defaults.get(option.dest)
         help_text = optparse.IndentedHelpFormatter.expand_default(self, option)
 
-        if default_value and option.metavar == 'URL':
-            help_text = help_text.replace(
-                default_value, redact_auth_from_url(default_value))
+        if default_values and option.metavar == 'URL':
+            if isinstance(default_values, string_types):
+                default_values = [default_values]
+
+            # If its not a list, we should abort and just return the help text
+            if not isinstance(default_values, list):
+                default_values = []
+
+            for val in default_values:
+                help_text = help_text.replace(
+                    val, redact_auth_from_url(val))
 
         return help_text
 
diff --git a/tests/functional/test_help.py b/tests/functional/test_help.py
index 9c2508abb51..a660cdf520d 100644
--- a/tests/functional/test_help.py
+++ b/tests/functional/test_help.py
@@ -74,6 +74,17 @@ def test_help_command_redact_auth_from_url(script):
     assert 'secret' not in result.stdout
 
 
+def test_help_command_redact_auth_from_url_with_extra_index_url(script):
+    """
+    Test `help` on various subcommands redact auth from url with extra index url
+    """
+    script.environ['PIP_INDEX_URL'] = 'https://user:secret@example.com'
+    script.environ['PIP_EXTRA_INDEX_URL'] = 'https://user:secret@example2.com'
+    result = script.pip('install', '--help')
+    assert result.returncode == SUCCESS
+    assert 'secret' not in result.stdout
+
+
 def test_help_commands_equally_functional(in_memory_pip):
     """
     Test if `pip help` and 'pip --help' behave the same way.


Code review comments:
diff_hunk: @@ -0,0 +1,2 @@
+Handle case of list default values in UpdatingDefaultsHelpFormatter
+Happens because we pass a list from --extra-index-url
path: news/9191.bugfix.rst
text: @user1:
```suggestion
Fix crash when logic for redacting authentication information from URLs
in ``--help`` is given a list of strings, instead of a single string.
```