Repository: facebookresearch/hydra

Issue:
[Bug] pdb support is broken
# 🐛 Bug
## Description

In patch #1239, debugger can work with hydra by setting `HYDRA_FULL_ERROR=1` , but pr #1465 breaks it. In #1465, `HydraJobException` is always raised if the job raise an exception, such that debugger will catch `HydraJobException` instead of the original exceptions.

## Checklist
- [x] I checked on the latest version of Hydra
- [ ] I created a minimal repro (See [this](https://stackoverflow.com/help/minimal-reproducible-example) for tips).

## To reproduce

** Minimal Code/Config snippet to reproduce **
```python
# mini.py
import hydra

@hydra.main()
def foo(cfg):
    raise Exception

foo()
```
then run `HYDRA_FULL_ERROR=1 python -m pdb -c continue mini.py` in shell.

pdb catch `HydraJobException` in `hydra/core/utils.py`

## Expected Behavior
pdb catch `Exception` in `foo`.

## System information
- **Hydra Version** : 1.1.0dev6
- **Python version** : 3.8.8
- **Virtual environment type and version** : 
- **Operating system** : linux 5.9.10

## Additional context

I think add `if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():` in `JobReturn.return_value` can temporarily fix this.
```python
@dataclass
class JobReturn:
    ....

    @property
    def return_value(self) -> Any:
        assert self.status != JobStatus.UNKNOWN, "return_value not yet available"
        if self.status == JobStatus.COMPLETED:
            return self._return_value
        else:
            if _is_env_set("HYDRA_FULL_ERROR") or is_under_debugger():  # add this
                raise self._return_value
            raise HydraJobException(
                f"Error executing job with overrides: {self.overrides}"
            ) from self._return_value
```


Discussion:
> In patch #1239, debugger can work with hydra by setting HYDRA_FULL_ERROR=1

Thanks for reporting.
#1239 enables pdb support without needing to set HYDRA_FULL_ERROR.
What happens if you do not set it?

Run `python mini.py`  has no problem. But if run `python -m pdb -c continue mini.py` or `HYDRA_FULL_ERROR=1 python -m pdb -c continue mini.py`, pdb will always catch a `HydraJobException` instead of the `Exception` raised in `mini.py`.

This is the output of pdb
```
/home/louchao/code/struct_vat2/mini.py:1: UserWarning: 
config_path is not specified in @hydra.main().
See https://hydra.cc/docs/next/upgrades/1.0_to_1.1/changes_to_hydra_main_config_path for more information.
  import hydra
Traceback (most recent call last):
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py", line 137, in run_job
    ret.return_value = task_function(task_cfg)
  File "/home/louchao/code/struct_vat2/mini.py", line 6, in foo
    raise Exception
Exception

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/louchao/miniconda3/lib/python3.8/pdb.py", line 1705, in main
    pdb._runscript(mainpyfile)
  File "/home/louchao/miniconda3/lib/python3.8/pdb.py", line 1573, in _runscript
    self.run(statement)
  File "/home/louchao/miniconda3/lib/python3.8/bdb.py", line 580, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/louchao/code/struct_vat2/mini.py", line 1, in <module>
    import hydra
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/main.py", line 49, in decorated_main
    _run_hydra(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 370, in _run_hydra
    run_and_report(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 215, in run_and_report
    raise ex
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 212, in run_and_report
    return func()
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/utils.py", line 371, in <lambda>
    lambda: hydra.run(
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 107, in run
    _ = ret.return_value
  File "/home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py", line 212, in return_value
    raise HydraJobException(
hydra.errors.HydraJobException: Error executing job with overrides: []
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> /home/louchao/miniconda3/lib/python3.8/site-packages/hydra/core/utils.py(212)return_value()
-> raise HydraJobException(
(Pdb) 
```

In #1239, `_is_env_set("HYDRA_FULL_ERROR") or is_under_debugger()` is checked, if `True`, `run_and_report` will throw the original exception directly. This is OK.

In #1465, the following change makes exception stacktrace changed before #1239. Let's focus on line 110 of `hydra/_internal/hydra.py`.
https://github.com/facebookresearch/hydra/blob/3f74e8fced2ae62f2098b701e7fdabc1eed3cbb6/hydra/_internal/hydra.py#L105-L112
which is calling
https://github.com/facebookresearch/hydra/blob/3f74e8fced2ae62f2098b701e7fdabc1eed3cbb6/hydra/core/utils.py#L226-L234
Here Hydra raise HydraJobException instead of the original exception even if under a debugger.

Resolving patch:
diff --git a/hydra/_internal/utils.py b/hydra/_internal/utils.py
--- a/hydra/_internal/utils.py
+++ b/hydra/_internal/utils.py
@@ -17,7 +17,6 @@
 from hydra.core.utils import get_valid_filename, validate_config_path
 from hydra.errors import (
     CompactHydraException,
-    HydraException,
     InstantiationException,
     SearchPathException,
 )
@@ -224,12 +223,6 @@ def run_and_report(func: Any) -> Any:
                 # It is possible to add additional libraries to sanitize from the bottom later,
                 # maybe even make it configurable.
 
-                # If got a HydraException that is caused by another exception,
-                # print the message of the HydraException and unwrap the cause.
-                if isinstance(ex, HydraException) and ex.__cause__ is not None:
-                    sys.stderr.write(str(ex) + os.linesep)
-                    ex = ex.__cause__  # type: ignore
-
                 tb: Any = ex.__traceback__
                 search_max = 10
                 # strip Hydra frames from start of stack
diff --git a/hydra/core/utils.py b/hydra/core/utils.py
--- a/hydra/core/utils.py
+++ b/hydra/core/utils.py
@@ -18,13 +18,11 @@
 
 from hydra.core.hydra_config import HydraConfig
 from hydra.core.singleton import Singleton
-from hydra.errors import HydraJobException
+from hydra.types import HydraContext, TaskFunction
 
 if TYPE_CHECKING:
     from hydra._internal.callbacks import Callbacks
 
-from hydra.types import HydraContext, TaskFunction
-
 log = logging.getLogger(__name__)
 
 
@@ -229,9 +227,10 @@ def return_value(self) -> Any:
         if self.status == JobStatus.COMPLETED:
             return self._return_value
         else:
-            raise HydraJobException(
-                f"Error executing job with overrides: {self.overrides}"
-            ) from self._return_value
+            sys.stderr.write(
+                f"Error executing job with overrides: {self.overrides}" + os.linesep
+            )
+            raise self._return_value
 
     @return_value.setter
     def return_value(self, value: Any) -> None:
diff --git a/hydra/errors.py b/hydra/errors.py
--- a/hydra/errors.py
+++ b/hydra/errors.py
@@ -39,9 +39,3 @@ def __init__(
         super(MissingConfigException, self).__init__(message)
         self.missing_cfg_file = missing_cfg_file
         self.options = options
-
-
-class HydraJobException(HydraException):
-    """Raised when accessing a result of a failed job."""
-
-    ...


Test patch:
diff --git a/tests/test_hydra.py b/tests/test_hydra.py
--- a/tests/test_hydra.py
+++ b/tests/test_hydra.py
@@ -1193,13 +1193,13 @@ def test_2(self) -> None:
         ),
     ],
 )
-def test_hydra_exception(
-    monkeypatch: Any,
+def test_job_exception(
     tmpdir: Any,
     expected: str,
 ) -> None:
-    monkeypatch.chdir("tests/test_apps/app_exception")
-    ret = run_with_error(["my_app.py", f"hydra.run.dir={tmpdir}"])
+    ret = run_with_error(
+        ["tests/test_apps/app_exception/my_app.py", f"hydra.run.dir={tmpdir}"]
+    )
     assert_regex_match(
         from_line=expected,
         to_line=ret,
@@ -1208,6 +1208,16 @@ def test_hydra_exception(
     )
 
 
+def test_job_exception_full_error(tmpdir: Any) -> None:
+    ret = run_with_error(
+        ["tests/test_apps/app_exception/my_app.py", f"hydra.run.dir={tmpdir}"],
+        env={**os.environ, "HYDRA_FULL_ERROR": "1"},
+    )
+
+    assert "ZeroDivisionError: division by zero" in ret
+    assert "Set the environment variable HYDRA_FULL_ERROR=1" not in ret
+
+
 def test_structured_with_none_list(monkeypatch: Any, tmpdir: Path) -> None:
     monkeypatch.chdir("tests/test_apps/structured_with_none_list")
     cmd = [