Repository: pypa/pip

Issue:
Allow troubleshooting with pdb
In https://github.com/pypa/pip/issues/9348#issuecomment-750930313, I attempted to use `pdb` to debug an unhandled exception in pip, but when I did, found that pdb couldn't intercept the exception because pip [handles all exceptions](https://github.com/pypa/pip/blob/f30a02c7c26446d0e2b1b986ecbbe402795f8ccd/src/pip/_internal/cli/base_command.py#L189-L227).

I suggest that pip should provide an alternate interface or (preferably) autodetect when pdb is involved and in this case not trap exceptions so that they might be intercepted by pdb.


Discussion:
In 3560f7735, I've drafted a change that implements the auto detection concept.

I then applied this diff:

```
$ git diff
diff --git a/src/pip/_internal/commands/install.py b/src/pip/_internal/commands/install.py
index a9a447872..6b161305e 100644
--- a/src/pip/_internal/commands/install.py
+++ b/src/pip/_internal/commands/install.py
@@ -225,6 +225,7 @@ class InstallCommand(RequirementCommand):
 
     @with_cleanup
     def run(self, options, args):
+        raise RuntimeError('trap me please')
         # type: (Values, List[str]) -> int
         if options.use_user_site and options.target_dir is not None:
             raise CommandError("Can not combine '--user' and '--target'")
```

to simulate an unhandled exception in the install command.

To demonstrate the effectiveness of the change, here's the old, default behavior:

```
pip feature/9349-nicer-debug $ .tox/python/bin/python -m pip install foo
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 196, in _trap_run
    status = self.run(options, args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 228, in run
    raise RuntimeError('trap me please')
RuntimeError: trap me please
```

And here's what happens when running under pdb:

```
pip feature/9349-nicer-debug $ .tox/python/bin/python -m pdb -m pip install foo
> /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> import os
(Pdb) continue
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/commands/install.py(228)run()
-> raise RuntimeError('trap me please')
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1702, in main
    pdb._runmodule(mainpyfile)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py", line 1547, in _runmodule
    self.run(code)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/__main__.py", line 1, in <module>
    import os
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/main.py", line 73, in main
    return command.main(cmd_args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 112, in main
    return self._main(args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 190, in _main
    status = runner(options, args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/req_command.py", line 178, in wrapper
    return func(self, options, args)
  File "/Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 228, in run
    raise RuntimeError('trap me please')
RuntimeError: trap me please
(Pdb) where
  /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py(1702)main()
-> pdb._runmodule(mainpyfile)
  /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/pdb.py(1547)_runmodule()
-> self.run(code)
  /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/bdb.py(580)run()
-> exec(cmd, globals, locals)
  /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> import os
  /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/main.py(73)main()
-> return command.main(cmd_args)
  /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/base_command.py(112)main()
-> return self._main(args)
  /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/base_command.py(190)_main()
-> status = runner(options, args)
  /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/cli/req_command.py(178)wrapper()
-> return func(self, options, args)
> /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/_internal/commands/install.py(228)run()
-> raise RuntimeError('trap me please')
(Pdb) quit
Post mortem debugger finished. The pip will be restarted
> /Users/jaraco/code/public/pypa/pip/.tox/python/lib/python3.9/site-packages/pip/__main__.py(1)<module>()
-> import os
(Pdb) 
```
Proposition given by @uranusjr that seems safer:

> I think our most reasonable approach is providing either an explicit debug entry point, or a flag (e.g. --debug) that disables exception trapping.

IMO `python -m pdb -m pip --debug` is the one with the least magic involved.



Resolving patch:
diff --git a/src/pip/_internal/cli/base_command.py b/src/pip/_internal/cli/base_command.py
--- a/src/pip/_internal/cli/base_command.py
+++ b/src/pip/_internal/cli/base_command.py
@@ -1,5 +1,6 @@
 """Base Command class, and related routines"""
 
+import functools
 import logging
 import logging.config
 import optparse
@@ -7,7 +8,7 @@
 import sys
 import traceback
 from optparse import Values
-from typing import List, Optional, Tuple
+from typing import Any, Callable, List, Optional, Tuple
 
 from pip._internal.cli import cmdoptions
 from pip._internal.cli.command_context import CommandContextMixIn
@@ -169,46 +170,60 @@ def _main(self, args: List[str]) -> int:
                 "This will become an error in pip 21.0."
             )
 
+        def intercepts_unhandled_exc(
+            run_func: Callable[..., int]
+        ) -> Callable[..., int]:
+            @functools.wraps(run_func)
+            def exc_logging_wrapper(*args: Any) -> int:
+                try:
+                    status = run_func(*args)
+                    assert isinstance(status, int)
+                    return status
+                except PreviousBuildDirError as exc:
+                    logger.critical(str(exc))
+                    logger.debug("Exception information:", exc_info=True)
+
+                    return PREVIOUS_BUILD_DIR_ERROR
+                except (
+                    InstallationError,
+                    UninstallationError,
+                    BadCommand,
+                    NetworkConnectionError,
+                ) as exc:
+                    logger.critical(str(exc))
+                    logger.debug("Exception information:", exc_info=True)
+
+                    return ERROR
+                except CommandError as exc:
+                    logger.critical("%s", exc)
+                    logger.debug("Exception information:", exc_info=True)
+
+                    return ERROR
+                except BrokenStdoutLoggingError:
+                    # Bypass our logger and write any remaining messages to
+                    # stderr because stdout no longer works.
+                    print("ERROR: Pipe to stdout was broken", file=sys.stderr)
+                    if level_number <= logging.DEBUG:
+                        traceback.print_exc(file=sys.stderr)
+
+                    return ERROR
+                except KeyboardInterrupt:
+                    logger.critical("Operation cancelled by user")
+                    logger.debug("Exception information:", exc_info=True)
+
+                    return ERROR
+                except BaseException:
+                    logger.critical("Exception:", exc_info=True)
+
+                    return UNKNOWN_ERROR
+
+            return exc_logging_wrapper
+
         try:
-            status = self.run(options, args)
-            assert isinstance(status, int)
-            return status
-        except PreviousBuildDirError as exc:
-            logger.critical(str(exc))
-            logger.debug("Exception information:", exc_info=True)
-
-            return PREVIOUS_BUILD_DIR_ERROR
-        except (
-            InstallationError,
-            UninstallationError,
-            BadCommand,
-            NetworkConnectionError,
-        ) as exc:
-            logger.critical(str(exc))
-            logger.debug("Exception information:", exc_info=True)
-
-            return ERROR
-        except CommandError as exc:
-            logger.critical("%s", exc)
-            logger.debug("Exception information:", exc_info=True)
-
-            return ERROR
-        except BrokenStdoutLoggingError:
-            # Bypass our logger and write any remaining messages to stderr
-            # because stdout no longer works.
-            print("ERROR: Pipe to stdout was broken", file=sys.stderr)
-            if level_number <= logging.DEBUG:
-                traceback.print_exc(file=sys.stderr)
-
-            return ERROR
-        except KeyboardInterrupt:
-            logger.critical("Operation cancelled by user")
-            logger.debug("Exception information:", exc_info=True)
-
-            return ERROR
-        except BaseException:
-            logger.critical("Exception:", exc_info=True)
-
-            return UNKNOWN_ERROR
+            if not options.debug_mode:
+                run = intercepts_unhandled_exc(self.run)
+            else:
+                run = self.run
+            return run(options, args)
         finally:
             self.handle_pip_version_check(options)
diff --git a/src/pip/_internal/cli/cmdoptions.py b/src/pip/_internal/cli/cmdoptions.py
--- a/src/pip/_internal/cli/cmdoptions.py
+++ b/src/pip/_internal/cli/cmdoptions.py
@@ -151,6 +151,18 @@ class PipOption(Option):
     help="Show help.",
 )
 
+debug_mode: Callable[..., Option] = partial(
+    Option,
+    "--debug",
+    dest="debug_mode",
+    action="store_true",
+    default=False,
+    help=(
+        "Let unhandled exceptions propagate outside the main subroutine, "
+        "instead of logging them to stderr."
+    ),
+)
+
 isolated_mode: Callable[..., Option] = partial(
     Option,
     "--isolated",
@@ -974,6 +986,7 @@ def check_list_path_option(options: Values) -> None:
     "name": "General Options",
     "options": [
         help_,
+        debug_mode,
         isolated_mode,
         require_virtualenv,
         verbose,


Test patch: