Repository: facebookresearch/hydra

Issue:
pdb can't be used if hydra main decorator used
# 🚀 Feature Request: PDB support

<!-- A clear and concise description of the feature you are requesting -->

## Motivation

I got used to debug pipelines with `pdb`, but it looks to be impossible with hydra decorator

### Example
Consider a script
```
# test.py
from omegaconf import DictConfig
import hydra


@hydra.main()
def main(cfg: DictConfig):
    1/0


if __name__ == '__main__':
    main()
```
and you want to use `pdb` to fix the error. For deep learning it might be inspecting weights for nans, grad norms, stack levels.
Without any interaction, program finishes with error code `1`

```
$ python test.py
Traceback (most recent call last):
  File "test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
```
with `HYDRA_FULL_ERROR=1` 
```
$ HYDRA_FULL_ERROR=1 python test.py
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
```
with python debugger `pdb` which is possible to catch any unexpected exception
```
$ HYDRA_FULL_ERROR=1 python -m pdb -c continue test.py
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
The program exited via sys.exit(). Exit status: 1
> /home/ferres/ntl/cfan/experiments/hydra/test.py(1)<module>()
-> from omegaconf import DictConfig
(Pdb) ll
  1  -> from omegaconf import DictConfig
  2     import hydra
  3     
  4     
  5     @hydra.main()
  6     def main(cfg: DictConfig):
  7         1/0
  8     
  9     
 10     if __name__ == '__main__':
 11         main()
(Pdb) 
```
but expected to be 
```
 HYDRA_FULL_ERROR=1 python -m pdb -c continue test.py                  
Traceback (most recent call last):
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
Traceback (most recent call last):
  File "/usr/lib/python3.8/pdb.py", line 1704, in main
    pdb._runscript(mainpyfile)
  File "/usr/lib/python3.8/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/usr/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 1, in <module>
    from omegaconf import DictConfig
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/main.py", line 32, in decorated_main
    _run_hydra(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 352, in _run_hydra
    run_and_report(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 276, in run_and_report
    raise ex
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 203, in run_and_report
    return func()
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/utils.py", line 353, in <lambda>
    lambda: hydra.run(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 105, in run
    return run_job(
  File "/home/ferres/ntl/venv/lib/python3.8/site-packages/hydra/core/utils.py", line 123, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/ferres/ntl/cfan/experiments/hydra/test.py", line 7, in main
    1/0
ZeroDivisionError: division by zero
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/ferres/ntl/cfan/experiments/hydra/test.py(7)main()
-> 1/0
(Pdb) ll
  5     @hydra.main()
  6     def main(cfg: DictConfig):
  7  ->     1/0

```
## Pitch

I see some possible solutions
A simple solution is to add a flag `HYDRA_DEBUG` to env variables as done with `HYDRA_FULL_TRACE`. [Here](https://github.com/facebookresearch/hydra/blob/adb5808d8e221847dd591874cacaa7fbe34b393d/hydra/_internal/utils.py#L276) Exception handling is too broad and after a clean up you can still decide between `sys.exit(1)` and `raise ex` (changing this locally solved my problems)

**Describe alternatives you've considered**
I considered manually hacking source code of `/hydra/_internal/utils.py` (1 line change)

**Are you willing to open a pull request?** (See [CONTRIBUTING](../../CONTRIBUTING.md)) <-- broken link in template
Yes, why not




Discussion:
1. Please set a title to your issue.
2. Please explain what the problem is before suggesting solutions.
@omry done
Thanks.

Resolving patch:
diff --git a/hydra/_internal/utils.py b/hydra/_internal/utils.py
--- a/hydra/_internal/utils.py
+++ b/hydra/_internal/utils.py
@@ -198,12 +198,16 @@ def create_config_search_path(search_path_dir: Optional[str]) -> ConfigSearchPat
     return search_path
 
 
+def _is_env_set(name: str) -> bool:
+    return name in os.environ and os.environ[name] == "1"
+
+
 def run_and_report(func: Any) -> Any:
     try:
         return func()
     except Exception as ex:
-        if "HYDRA_FULL_ERROR" in os.environ and os.environ["HYDRA_FULL_ERROR"] == "1":
-            print_exc()
+        if _is_env_set("HYDRA_FULL_ERROR"):
+            raise ex
         else:
             if isinstance(ex, CompactHydraException):
                 sys.stderr.write(str(ex) + os.linesep)


Test patch:
diff --git a/hydra/test_utils/test_utils.py b/hydra/test_utils/test_utils.py
--- a/hydra/test_utils/test_utils.py
+++ b/hydra/test_utils/test_utils.py
@@ -359,7 +359,7 @@ def run_with_error(cmd: Any, env: Any = None) -> str:
     with Popen(cmd, stdout=PIPE, stderr=PIPE, env=env) as p:
         _stdout, stderr = p.communicate()
         err = stderr.decode("utf-8").rstrip().replace("\r\n", "\n")
-        assert p.returncode == 1
+        assert p.returncode != 0
     return err
 
 
@@ -383,22 +383,24 @@ def normalize_newlines(s: str) -> str:
     return s.replace("\r\n", "\n").replace("\r", "\n")
 
 
-def assert_text_same(actual: str, expected: str) -> None:
-    actual = normalize_newlines(actual)
-    expected = normalize_newlines(expected)
+def assert_text_same(
+    from_line: str, to_line: str, from_name: str = "Expected", to_name: str = "Actual"
+) -> None:
+    from_line = normalize_newlines(from_line)
+    to_line = normalize_newlines(to_line)
     lines = [
         line
         for line in unified_diff(
-            a=actual.splitlines(),
-            b=expected.splitlines(),
-            fromfile="Expected",
-            tofile="Actual",
+            a=from_line.splitlines(),
+            b=to_line.splitlines(),
+            fromfile=from_name,
+            tofile=to_name,
         )
     ]
 
     diff = "\n".join(lines)
     if len(diff) > 0:
-        print("------------ DIFF -------------")
+        print("\n------------ DIFF -------------")
         print(diff)
         print("-------------------------------")
         assert False, "Mismatch between expected and actual text"
diff --git a/tests/test_apps/app_exception/__init__.py b/tests/test_apps/app_exception/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_exception/__init__.py
@@ -0,0 +1 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
diff --git a/tests/test_apps/app_exception/my_app.py b/tests/test_apps/app_exception/my_app.py
new file mode 100644
--- /dev/null
+++ b/tests/test_apps/app_exception/my_app.py
@@ -0,0 +1,13 @@
+# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved
+from omegaconf import DictConfig
+
+import hydra
+
+
+@hydra.main()
+def my_app(_: DictConfig) -> None:
+    1 / 0
+
+
+if __name__ == "__main__":
+    my_app()
diff --git a/tests/test_examples/test_configure_hydra.py b/tests/test_examples/test_configure_hydra.py
--- a/tests/test_examples/test_configure_hydra.py
+++ b/tests/test_examples/test_configure_hydra.py
@@ -48,7 +48,7 @@ def test_custom_help(tmpdir: Path) -> None:
             Use --hydra-help to view Hydra specific help
 """
     )
-    assert_text_same(actual=result, expected=expected)
+    assert_text_same(from_line=expected, to_line=result)
 
 
 def test_job_name_no_config_override(tmpdir: Path) -> None:
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -5,6 +5,7 @@
 import sys
 from logging import getLogger
 from pathlib import Path
+from textwrap import dedent
 from typing import Any, List, Optional, Set
 
 import pytest
@@ -14,6 +15,7 @@
 from hydra.test_utils.test_utils import (
     TSweepRunner,
     TTaskRunner,
+    assert_text_same,
     chdir_hydra_root,
     get_run_output,
     integration_test,
@@ -1099,3 +1101,32 @@ def test_1(
     def test_2(self) -> None:
         logger = getLogger(__name__)
         logger.info("test_2")
+
+
+@pytest.mark.parametrize(  # type: ignore
+    "expected",
+    [
+        (
+            dedent(
+                """\
+                Traceback (most recent call last):
+                  File "my_app.py", line 9, in my_app
+                    1 / 0
+                ZeroDivisionError: division by zero
+
+                Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
+                """
+            )
+        ),
+    ],
+)
+def test_hydra_exception(monkeypatch: Any, tmpdir: Any, expected: str) -> None:
+    monkeypatch.chdir("tests/test_apps/app_exception")
+    ret = run_with_error(["my_app.py", f"hydra.run.dir={tmpdir}"])
+
+    assert_text_same(
+        from_line=expected,
+        to_line=ret,
+        from_name="Expected output",
+        to_name="Actual output",
+    )