[Autograd] Queued Callback Does Not Propagate Error
Labels: module: autograd, triaged, actionable

When queueing a callback using `torch.autograd.Variable._execution_engine.queue_callback()` for a GPU tensor, a raised error in the callback is not propagated. Instead, we see a message like `SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7fa8f8172850> returned NULL without setting an error`.

<details>
  <summary>Reproducer</summary>
  
  ```python
import torch

def callback():
    print("Callback! Raising an error...")
    raise RuntimeError("Error from callback!")

def hook_with_callback(*args):
    print("Backward hook!")
    torch.autograd.Variable._execution_engine.queue_callback(callback)

t = torch.tensor([1., 2.], requires_grad=True, device=torch.device("cuda"))
t.register_hook(hook_with_callback)
output = t ** 2
loss = output.sum()
loss.backward()
  ```
  
</details>

<details>
  <summary>Output</summary>
  
  ```shell
Backward hook!
Callback! Raising an error...
Traceback (most recent call last):
  File "repro.py", line 15, in <module>
    loss.backward()
  File "/fsx/users/andgu/work/pytorch/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/fsx/users/andgu/work/pytorch/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7fa8f8172850> returned NULL without setting an error
  ```

</details>

If the tensor `t` is on CPU instead, then the error is correctly propagated.

</details>

<details>
  <summary>Output when tensor is instead on CPU</summary>
  
  ```shell
Backward hook!
Callback! Raising an error...
Traceback (most recent call last):
  File "repro.py", line 15, in <module>
    loss.backward()
  File "/fsx/users/andgu/work/pytorch/torch/_tensor.py", line 396, in backward
    torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)
  File "/fsx/users/andgu/work/pytorch/torch/autograd/__init__.py", line 173, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
  File "repro.py", line 5, in callback
    raise RuntimeError("Error from callback!")
RuntimeError: Error from callback!
  ```

</details>

I am not sure if there are constraints regarding the tensor being on GPU that make this difficult. Both PyTorch FSDP and Fairscale FSDP have a `p_assert()` method (https://github.com/pytorch/pytorch/blob/d4cce30573651b5683d7e932b88194425db23758/torch/distributed/fsdp/fully_sharded_data_parallel.py#L2872 and [here in Fairscale](https://github.com/facebookresearch/fairscale/blob/72f373c1ac00e6a0506fdecbeed5bb1857e98ce7/fairscale/nn/data_parallel/fully_sharded_data_parallel.py#L2422)) to ensure the error message is printed before erroring. However, this has previously been used for internal assertions only.

The downside of using `p_assert()` (or something similar that prints before erroring) is that the cryptic message  `SystemError: <built-in method run_backward of torch._C._EngineBase object at 0x7fa8f8172850> returned NULL without setting an error` is still written to console, which may confuse users.

cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7