Dynamo minifier fails with false internal assert on torch-nightly
Labels: triaged, oncall: pt2

### 🐛 Describe the bug

This issue came up when I tried to debug #91710, so the setup is the same.

I'm running a modified Stable Diffusion code with `@torch.compile` [applied on one of the functions](https://github.com/sgrigory/stablediffusion2/blob/115130e4a9c8562225f24e6929575906752c79f9/ldm/models/diffusion/plms.py#L187). The compilation breaks with `TypeError: object of type 'tuple_iterator' has no len()` (separate issue #91710).
After adding a minifier with `TORCHDYNAMO_REPRO_AFTER="dynamo"` I'm getting `RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus1256814507.`

Note:
- The error doesn't go away after setting `torch._inductor.config.triton.cudagraphs = False`
- The full code is [here](https://github.com/sgrigory/stablediffusion2/tree/115130e4a9c8562225f24e6929575906752c79f9)

@msaroufim
<details>
  <summary>Show full error log</summary>

```
$ TORCHDYNAMO_REPRO_AFTER="dynamo" with-proxy python stable-diffusion/scripts/txt2img.py --prompt "A photo" --seed 187 --plms --config stable-diffusion/configs/stable-diffusion/v2-inference_native_mha.yaml --ckpt /tmp/model_native_mha.ckpt --n_iter 1 --n_samples 1
------------------------------------------------------

Global seed set to 187
Loading model from /tmp/model_native_mha.ckpt
Global Step: 875000
No module 'xformers'. Proceeding without it.
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 865.99 M params.
making attention of type 'vanilla' with 512 in_channels
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
making attention of type 'vanilla' with 512 in_channels
Creating invisible watermark encoder (see https://github.com/ShieldMnt/invisible-watermark)...
Sampling:   0%|                                                                                                                                             | 0/1 [00:00<?, ?it/sData shape for PLMS sampling is (1, 4, 64, 64)                                                                                                               | 0/1 [00:00<?, ?it/s]
Running PLMS Sampling with 50 timesteps
                                                                                                                                                                                 [2023-01-04 01:54:32,224] torch._dynamo.debug_utils: [WARNING] Compiled Fx GraphModule failed. Creating script to minify the error.
[2023-01-04 01:54:32,227] torch._dynamo.debug_utils: [WARNING] Writing minified repro to /data/sandcastle/boxes/fbsource/torch_compile_debug/run_2023_01_04_01_54_32_226774/minifier/minifier_launcher.py
PLMS Sampler:   0%|                                                                                                                                        | 0/50 [00:01<?, ?it/s]
data:   0%|                                                                                                                                                 | 0/1 [00:02<?, ?it/s]
Sampling:   0%|                                                                                                                                             | 0/1 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 77, in preserve_rng_state
    yield
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2052, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_tensor_args, aot_config)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 1307, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 957, in aot_dispatch_base
    compiled_fw = aot_config.fw_compiler(fw_module, flat_args)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 88, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 373, in fw_compiler
    return inner_compile(
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 588, in debug_wrapper
    compiled_fn = compiler_fn(gm, example_inputs, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/debug.py", line 223, in inner
    return fn(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 140, in compile_fx_inner
    compiled_fn = graph.compile_to_fn()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/graph.py", line 538, in compile_to_fn
    return self.compile_to_module().call
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 88, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/graph.py", line 527, in compile_to_module
    mod = PyCodeCache.load(code)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 468, in load
    exec(code, mod.dict, mod.dict)
  File "/tmp/torchinductor_grigorysizov/hx/chxvn6a46wyylpjtjd4dhohms25zpkk4b4nakhhnykqyqkn6qcbk.py", line 131, in <module>
    async_compile.wait(globals())
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 663, in wait
    scope[key] = result.result()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 522, in result
    kernel = self.kernel = _load_kernel(self.source_code)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/codecache.py", line 502, in _load_kernel
    kernel.precompile()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py", line 59, in precompile
    self.launchers = [
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py", line 60, in <listcomp>
    self._precompile_config(c, warm_cache_only_with_cc)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/triton_ops/autotune.py", line 84, in _precompile_config
    torch.cuda.synchronize(torch.cuda.current_device())
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/cuda/init.py", line 577, in synchronize
    return torch._C._cuda_synchronize()
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 680, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/debug_utils.py", line 1006, in debug_wrapper
    compiled_gm = compiler_fn(
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/init.py", line 1190, in call
    return self.compile_fn(model_, inputs_)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_inductor/compile_fx.py", line 398, in compile_fx
    return aot_autograd(
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/optimizations/training.py", line 78, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2355, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 88, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 2057, in create_aot_dispatcher_function
    return compiled_fn
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/contextlib.py", line 131, in exit
    self.gen.throw(type, value, traceback)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_functorch/aot_autograd.py", line 81, in preserve_rng_state
    torch.cuda.set_rng_state(cuda_rng_state)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/cuda/random.py", line 64, in set_rng_state
    _lazy_call(cb)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/cuda/init.py", line 176, in _lazy_call
    callable()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/cuda/random.py", line 62, in cb
    default_generator.set_state(new_state_copy)
RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus1588869035

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "stable-diffusion/scripts/txt2img.py", line 325, in <module>
    main(opt)
  File "stable-diffusion/scripts/txt2img.py", line 275, in main
    samples, _ = sampler.sample(S=opt.steps,
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
    return func(*args, **kwargs)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/plms.py", line 99, in sample
    samples, intermediates = self.plms_sampling(conditioning, size,
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
    return func(*args, **kwargs)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/plms.py", line 167, in plms_sampling
    outs = self.p_sample_plms(img, cond, ts, index=index, use_original_steps=ddim_use_original_steps,
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 212, in _fn
    return fn(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
    return func(*args, **kwargs)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/plms.py", line 232, in p_sample_plms
    e_t = get_model_output(x, t)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/plms.py", line 202, in get_model_output
    e_t_uncond, e_t = self.model.apply_model(x_in, t_in, c_in).chunk(2)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/ddpm.py", line 858, in apply_model
    x_recon = self.model(x_noisy, t, **cond)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/models/diffusion/ddpm.py", line 1329, in forward
    out = self.diffusion_model(x, t, context=cc)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl
    return forward_call(*args, **kwargs)
  File "/data/sandcastle/boxes/fbsource/stable-diffusion/ldm/modules/diffusionmodules/openaimodel.py", line 770, in forward
    emb = self.time_embed(t_emb)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1482, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 333, in catch_errors
    return callback(frame, cache_size, hooks)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 480, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 103, in _fn
    return fn(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 88, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 339, in _convert_frame_assert
    return _compile(
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 400, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/bytecode_transformation.py", line 341, in transform_code_object
    transformations(instructions, code_options)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 387, in transform
    tracer.run()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1684, in run
    super().run()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 538, in run
    and self.step()
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 501, in step
    getattr(self, inst.opname)(inst)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/symbolic_convert.py", line 1750, in RETURN_VALUE
    self.output.compile_subgraph(self)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 533, in compile_subgraph
    self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 604, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/grigorysizov/anaconda3/envs/ldm/lib/python3.8/site-packages/torch/_dynamo/output_graph.py", line 685, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: debug_wrapper raised RuntimeError: false INTERNAL ASSERT FAILED at "../c10/cuda/CUDAGraphsC10Utils.h":73, please report a bug to PyTorch. Unknown CUDA graph CaptureStatus1588869035

Set torch._dynamo.config.verbose=True for more information

Minifier script written to /data/sandcastle/boxes/fbsource/torch_compile_debug/run_2023_01_04_01_54_32_226774/minifier/minifier_launcher.py. Run this script to find the smallest traced graph which reproduces this error.


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
```
</details>

### Versions

```
PyTorch version: 2.0.0.dev20230102+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A


OS: CentOS Stream 8 (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-17)
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.28


Python version: 3.8.5 (default, Sep  4 2020, 07:30:14)  [GCC 7.3.0] (64-bit runtime)
Python platform: Linux-5.12.0-0_fbk9_clang_7032_gb6cf154957d9-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: Tesla V100-SXM2-16GB
GPU 1: Tesla V100-SXM2-16GB
GPU 2: Tesla V100-SXM2-16GB
GPU 3: Tesla V100-SXM2-16GB
GPU 4: Tesla V100-SXM2-16GB
GPU 5: Tesla V100-SXM2-16GB
GPU 6: Tesla V100-SXM2-16GB
GPU 7: Tesla V100-SXM2-16GB


Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True


Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] open-clip-torch==2.0.2
[pip3] pytorch-lightning==1.4.2
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] torch==2.0.0.dev20230102+cu117
[pip3] torchmetrics==0.6.0
[pip3] torchvision==0.15.0.dev20230102+cu117
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.4.1               h8ab8bb3_9    nvidia
[conda] mkl                       2021.4.0           h06a4308_640  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] numpy                     1.23.1           py38h6c91a56_0  
[conda] numpy-base                1.23.1           py38ha15fc14_0  
[conda] open-clip-torch           2.0.2                    pypi_0    pypi
[conda] pytorch-lightning         1.4.2                    pypi_0    pypi
[conda] pytorch-triton            2.0.0+0d7e753227          pypi_0    pypi
[conda] torch                     2.0.0.dev20230102+cu117          pypi_0    pypi
[conda] torchmetrics              0.6.0                    pypi_0    pypi
[conda] torchvision               0.15.0.dev20230102+cu117          pypi_0    pypi
```


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh