`torch.compile` frees computation graph in a GAN training setup and tries to call `backward` a second time
Labels: module: autograd, triaged, oncall: pt2, module: aotdispatch

### 🐛 Describe the bug

Reported by @edeyneka in the discussion board in [this topic](https://discuss.pytorch.org/t/adversarial-training-with-torch-compile/168933).
Ekaterina used the [DCGAN tutorial](https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html) and provided a minimal, executable code snippet to reproduce the issue [here](https://discuss.pytorch.org/t/adversarial-training-with-torch-compile/168933/3?u=ptrblck) (thanks again for providing the code snippet and reporting the issue).

A smaller code snippet is attached:
```python
import torch
import torch.nn as nn


def train(netD, netG, x):
    # train netD
    fake = netG(x)
    outD_only = netD(fake.detach())
    outD_only.mean().backward()
    
    # confirm the backward call does not touch netG
    print(["{}, {}".format(name, p.grad) for name, p in netG.named_parameters()])
    
    # train netG
    outD_attached = netD(fake)
    outD_attached.mean().backward()
    
    # confirm netG now has valid gradients
    print(["{}, {}".format(name, p.grad) for name, p in netG.named_parameters()])


## Eager
# setup
netG = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

netD = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

x = torch.randn(1, 1)

train(netD, netG, x)
# ['0.weight, None', '0.bias, None', '2.weight, None', '2.bias, None']
# ['0.weight, tensor([[6.7547e-05]])', '0.bias, tensor([0.0002])', '2.weight, tensor([[0.0008]])', '2.bias, tensor([0.0009])']

   
## Compile
# setup
netG = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

netD = nn.Sequential(
    nn.Linear(1, 1),
    nn.ReLU(),
    nn.Linear(1, 1))

x = torch.randn(1, 1)
train_compiled = torch.compile(train, mode="default")

train_compiled(netD, netG, x)
# ['0.weight, tensor([[0.]])', '0.bias, tensor([0.])', '2.weight, tensor([[0.]])', '2.bias, tensor([0.])'] # grads seems to be already created
# RuntimeError: Trying to backward through the graph a second time (or directly access saved tensors after they have already been freed). Saved intermediate values of the graph are freed when you call .backward() or autograd.grad(). Specify retain_graph=True if you need to backward through the graph a second time or if you need to access saved tensors after calling backward.
```
which simulates a GAN training routine.

The expected logic would be:
* train the discriminator using a real sample (let's ignore this forward/backward pass here as it's not failing)
* train the discriminator using a detached fake sample (created by the generator)
* the `outD_only.mean().backward()` should not touch `netG` at all since `fake` was detached in `outD_only = netD(fake.detach())`
* the eager mode execution also verifies this assumption by printing `None` gradients in `netG`
* train the generator by using the attached `fake` tensor in `netD`
* this works in eager mode but fails when `torch.compile` is used with the "Trying to backward a second time" error.



### Versions

PyTorch version: 2.0.0.dev20221227+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.31

Python version: 3.8.15 | packaged by conda-forge | (default, Nov 22 2022, 08:49:35)  [GCC 10.4.0] (64-bit runtime)
Python platform: Linux-5.13.0-41-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3090
Nvidia driver version: 515.43.04
cuDNN version: Probably one of the following:
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.5.0
/usr/local/cuda-11.7/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.5.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.24.1
[pip3] numpydoc==1.5.0
[pip3] torch==2.0.0.dev20221227+cu118
[pip3] torchtriton==2.0.0+0d7e753227
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] numpydoc                  1.5.0                    pypi_0    pypi
[conda] torch                     2.0.0.dev20221227+cu118          pypi_0    pypi
[conda] torchtriton               2.0.0+0d7e753227          pypi_0    pypi


cc @ezyang @albanD @zou3519 @gqchen @pearu @nikitaved @soulitzer @Lezcano @Varal7 @soumith @msaroufim @wconstab @ngimel @bdhirsh