PyTorch 2.0: AttributeError: __torch__.torch.classes.c10d.ProcessGroup (of Python compilation unit at: 0) does not have a field with name 'shape'
Labels: oncall: distributed, triaged, oncall: pt2

### 🐛 Describe the bug

I obtained the following traceback when enabling `torch.compile`:

```bash
/home/user/.local/lib/python3.10/site-packages/torch/fx/graph.py:1370: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
/home/user/.local/lib/python3.10/site-packages/torch/fx/graph.py:1370: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
/home/user/.local/lib/python3.10/site-packages/torch/fx/graph.py:1370: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
/home/user/.local/lib/python3.10/site-packages/torch/fx/graph.py:1370: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
  warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
Traceback (most recent call last):
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 692, in call_user_compiler
    compiled_fn = compiler_fn(gm, self.fake_example_inputs())
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/optimizations/distributed.py", line 201, in compile_fn
    return self.backend_compile_fn(gm, example_inputs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 1048, in debug_wrapper
    compiled_gm = compiler_fn(gm, example_inputs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/__init__.py", line 1325, in __call__
    return self.compile_fn(model_, inputs_)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/optimizations/backends.py", line 24, in inner
    return fn(gm, example_inputs, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/optimizations/backends.py", line 61, in inductor
    return compile_fx(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 418, in compile_fx
    return aot_autograd(
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/optimizations/training.py", line 74, in compiler_fn
    cg = aot_module_simplified(gm, example_inputs, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2483, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 162, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2180, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1411, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1688, in aot_dispatch_autograd
    compiled_fw_func = aot_config.fw_compiler(
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 162, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 393, in fw_compiler
    return inner_compile(
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/debug_utils.py", line 588, in debug_wrapper
    compiled_fn = compiler_fn(gm, example_inputs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/debug.py", line 239, in inner
    return fn(*args, **kwargs)
  File "/gpfslocalsup/pub/anaconda-py3/2021.05/envs/python-3.10.4/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 155, in compile_fx_inner
    graph.run(*example_inputs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 162, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/graph.py", line 181, in run
    return super().run(*args)
  File "/home/user/.local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 136, in run
    self.env[node] = self.run_node(node)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/graph.py", line 391, in run_node
    result = super().run_node(n)
  File "/home/user/.local/lib/python3.10/site-packages/torch/fx/interpreter.py", line 177, in run_node
    return getattr(self, n.op)(n.target, args, kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_inductor/graph.py", line 327, in get_attr
    if value.shape == ():
AttributeError: __torch__.torch.classes.c10d.ProcessGroup (of Python compilation unit at: 0) does not have a field with name 'shape'

While executing %_tensor_constant0 : [#users=1] = get_attr[target=_tensor_constant0]
Original traceback:
None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/code/bert/main_quantize.py", line 140, in <module>
    main(args)
  File "/home/user/code/bert/main_quantize.py", line 134, in main
    training_loop(args, model, device, data_loader_train, data_loader_val,
  File "/home/user/code/bert/common.py", line 333, in training_loop
    train_stats = train_fn(args, model, device, data_loader_train,
  File "/home/user/code/bert/engine_quantize.py", line 64, in train_epoch
    outputs = model(images, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1157, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 1111, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 211, in _fn
    return fn(*args, **kwargs)
  File "/home/user/code/bert/models/quantizers.py", line 501, in forward
    with torch.cuda.amp.autocast(enabled=False):
  File "/home/user/code/bert/models/quantizers.py", line 502, in <graph break in forward>
    output = self.quantizer(z.float(), **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1488, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 329, in catch_errors
    return hijacked_callback(frame, cache_size, hooks)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 403, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 103, in _fn
    return fn(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 261, in _convert_frame_assert
    return _compile(
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 162, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 323, in _compile
    out_code = transform_code_object(code, transform)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/bytecode_transformation.py", line 339, in transform_code_object
    transformations(instructions, code_options)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 310, in transform
    tracer.run()
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1715, in run
    super().run()
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 564, in run
    and self.step()
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 527, in step
    getattr(self, inst.opname)(inst)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 1781, in RETURN_VALUE
    self.output.compile_subgraph(self)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 563, in compile_subgraph
    self.compile_and_call_fx_graph(tx, pass2.graph_output_vars(), root)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 610, in compile_and_call_fx_graph
    compiled_fn = self.call_user_compiler(gm)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 162, in time_wrapper
    r = func(*args, **kwargs)
  File "/home/user/.local/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 697, in call_user_compiler
    raise BackendCompilerFailed(self.compiler_fn, e) from e
torch._dynamo.exc.BackendCompilerFailed: compile_fn raised AttributeError: __torch__.torch.classes.c10d.ProcessGroup (of Python compilation unit at: 0) does not have a field with name 'shape'

While executing %_tensor_constant0 : [#users=1] = get_attr[target=_tensor_constant0]
Original traceback:
None

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True
```

Adding `TORCHDYNAMO_REPRO_AFTER="dynamo"` at the beginning of the command, I got:

```bash
[2023-02-10 16:20:49,259] torch._dynamo.debug_utils: [WARNING] Compiled Fx GraphModule failed. Creating script to minify the error.
[2023-02-10 16:20:49,276] torch._dynamo.debug_utils: [WARNING] Writing minified repro to /home/user/code/bert/torch_compile_debug/run_2023_02_10_16_20_49_261548/minifier/minifier_launcher.py
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Exception raised from c10_cuda_check_implementation at ../c10/cuda/CUDAException.cpp:44 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x57 (0x14a3a3ed0b77 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&) + 0x64 (0x14a3a3e9a383 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x118 (0x14a3a3f7d9e8 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10_cuda.so)
frame #3: <unknown function> + 0x10684f5 (0x14a3a50ab4f5 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so)
frame #4: <unknown function> + 0x4cffe6 (0x14a3f94f4fe6 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #5: <unknown function> + 0x3f027 (0x14a3a3eb6027 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #6: c10::TensorImpl::~TensorImpl() + 0x1be (0x14a3a3eae85e in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #7: c10::TensorImpl::~TensorImpl() + 0x9 (0x14a3a3eae979 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #8: <unknown function> + 0x755838 (0x14a3f977a838 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #9: THPVariable_subclass_dealloc(_object*) + 0x305 (0x14a3f977abc5 in /home/user/.local/lib/python3.10/site-packages/torch/lib/libtorch_python.so)
frame #10: <unknown function> + 0x121b1b (0x55cc4bd8fb1b in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #11: <unknown function> + 0x12f1ba (0x55cc4bd9d1ba in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #12: <unknown function> + 0x21f97f (0x55cc4be8d97f in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #13: _PyEval_EvalFrameDefault + 0x5db8 (0x55cc4bda7678 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #14: _PyFunction_Vectorcall + 0x6f (0x55cc4bdb23cf in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #15: PyObject_Call + 0xb8 (0x55cc4bdc0348 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #16: _PyEval_EvalFrameDefault + 0x2b05 (0x55cc4bda43c5 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #17: _PyFunction_Vectorcall + 0x6f (0x55cc4bdb23cf in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #18: _PyEval_EvalFrameDefault + 0x1311 (0x55cc4bda2bd1 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #19: _PyFunction_Vectorcall + 0x6f (0x55cc4bdb23cf in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #20: _PyEval_EvalFrameDefault + 0x30c (0x55cc4bda1bcc in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #21: <unknown function> + 0x1eab92 (0x55cc4be58b92 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #22: PyEval_EvalCode + 0x87 (0x55cc4be58ad7 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #23: <unknown function> + 0x21d709 (0x55cc4be8b709 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #24: <unknown function> + 0x2183f4 (0x55cc4be863f4 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #25: <unknown function> + 0x982ba (0x55cc4bd062ba in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #26: _PyRun_SimpleFileObject + 0x1af (0x55cc4be805cf in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #27: _PyRun_AnyFileObject + 0x43 (0x55cc4be803a3 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #28: Py_RunMain + 0x39f (0x55cc4be7d56f in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #29: Py_BytesMain + 0x39 (0x55cc4be4be09 in /anaconda//envs/python-3.10.4/bin/python3.10)
frame #30: __libc_start_main + 0xf3 (0x14a43045dcf3 in /lib64/libc.so.6)
frame #31: <unknown function> + 0x1ddd11 (0x55cc4be4bd11 in /anaconda//envs/python-3.10.4/bin/python3.10)
```

And the minified script is the following:
```python

import os
from math import inf
import torch
from torch import tensor, device
import torch.fx as fx
import functools
import torch._dynamo
from torch._dynamo.debug_utils import run_fwd_maybe_bwd
from torch._dynamo.optimizations.backends import BACKENDS
from torch._dynamo.testing import rand_strided

import torch._dynamo.config
import torch._inductor.config
torch._dynamo.config.load_config(b'\x80\x04\x95\x16\x08\x00\x00\x00\x00\x00\x00}\x94(\x8c\x08__name__\x94\x8c\x14torch._dynamo.config\x94\x8c\x07__doc__\x94N\x8c\x0b__package__\x94\x8c\rtorch._dynamo\x94\x8c\n__loader__\x94\x8c\x1a_frozen_importlib_external\x94\x8c\x10SourceFileLoader\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94h\x02\x8c\x04path\x94\x8c[/linkhome/rech/gensak01/ugt86fu/.local/lib/python3.10/site-packages/torch/_dynamo/config.py\x94ub\x8c\x08__spec__\x94\x8c\x11_frozen_importlib\x94\x8c\nModuleSpec\x94\x93\x94)\x81\x94}\x94(h\x0ch\x02\x8c\x06loader\x94h\n\x8c\x06origin\x94h\x0e\x8c\x0cloader_state\x94N\x8c\x1asubmodule_search_locations\x94N\x8c\r_set_fileattr\x94\x88\x8c\x07_cached\x94\x8ct/linkhome/rech/gensak01/ugt86fu/.local/lib/python3.10/site-packages/torch/_dynamo/__pycache__/config.cpython-310.pyc\x94\x8c\r_initializing\x94\x89ub\x8c\x08__file__\x94h\x0e\x8c\n__cached__\x94h\x1b\x8c\x07abspath\x94\x8c\tposixpath\x94h\x1f\x93\x94\x8c\x07dirname\x94h h"\x93\x94\x8c\tlog_level\x94K\x1e\x8c\x0boutput_code\x94\x89\x8c\rlog_file_name\x94N\x8c\x07verbose\x94\x89\x8c\x11output_graph_code\x94\x89\x8c\x12verify_correctness\x94\x89\x8c\x12minimum_call_count\x94K\x01\x8c\x15dead_code_elimination\x94\x88\x8c\x10cache_size_limit\x94K@\x8c\x14specialize_int_float\x94\x88\x8c\x0edynamic_shapes\x94\x89\x8c\x10guard_nn_modules\x94\x89\x8c\x0cnormalize_ir\x94\x89\x8c\x1btraceable_tensor_subclasses\x94\x8f\x94\x8c\x0fsuppress_errors\x94\x89\x8c\x15replay_record_enabled\x94\x89\x8c rewrite_assert_with_torch_assert\x94\x88\x8c\x12print_graph_breaks\x94\x89\x8c\x07disable\x94\x89\x8c*allowed_functions_module_string_ignorelist\x94\x8f\x94(\x8c\x0btorch._refs\x94\x8c\rtorch.testing\x94\x8c\x13torch.distributions\x94\x8c\rtorch._decomp\x94\x8c\x0ctorch._prims\x94\x90\x8c\x0frepro_tolerance\x94G?PbM\xd2\xf1\xa9\xfc\x8c\x16capture_scalar_outputs\x94\x89\x8c\x19enforce_cond_guards_match\x94\x88\x8c\x0coptimize_ddp\x94\x88\x8c\x1araise_on_ctx_manager_usage\x94\x88\x8c\x1craise_on_unsafe_aot_autograd\x94\x89\x8c\rdynamo_import\x94\x8c\rtorch._dynamo\x94\x8c\x0finductor_import\x94\x8c\x0ftorch._inductor\x94\x8c\x18error_on_nested_fx_trace\x94\x88\x8c\tallow_rnn\x94\x89\x8c\x08base_dir\x94\x8cC/linkhome/rech/gensak01/ugt86fu/.local/lib/python3.10/site-packages\x94\x8c\x0edebug_dir_root\x94\x8cJ/gpfsdswork/projects/rech/nph/ugt86fu/code/visionBERT2/torch_compile_debug\x94\x8c)DO_NOT_USE_legacy_non_fake_example_inputs\x94\x89\x8c\x15_AccessLimitingConfig\x94}\x94(\x8c\n__module__\x94h\x02\x8c\x0b__setattr__\x94h\x02\x8c!_AccessLimitingConfig.__setattr__\x94\x93\x94h\x03Nu\x8c\x15_allowed_config_names\x94\x8f\x94(\x8c\x0brepro_after\x94\x8c\x0eexternal_utils\x94h?\x8c\nModuleType\x94h0h)\x8c\x12constant_functions\x94h.hEhOh\x1fh,h/h\x1ehJh%hC\x8c\x0brepro_level\x94hPh\x01h"\x8c\x03sys\x94h8h+hMh*hD\x8c!skipfiles_inline_module_allowlist\x94\x8c\x07logging\x94hIhB\x8c\x02os\x94h5\x8c\x05torch\x94h$\x8c\x0c__builtins__\x94hKh\x06h\x04h@h-h6h1h\'h\x03h7hAh\x1dhGh&h(h\x0fh4h3\x90\x8c\x1cget_config_serialization_fns\x94\x8c\x1atorch._dynamo.config_utils\x94hc\x93\x94u.')
torch._inductor.config.load_config(b'\x80\x04\x95G\t\x00\x00\x00\x00\x00\x00}\x94(\x8c\x08__name__\x94\x8c\x16torch._inductor.config\x94\x8c\x07__doc__\x94N\x8c\x0b__package__\x94\x8c\x0ftorch._inductor\x94\x8c\n__loader__\x94\x8c\x1a_frozen_importlib_external\x94\x8c\x10SourceFileLoader\x94\x93\x94)\x81\x94}\x94(\x8c\x04name\x94h\x02\x8c\x04path\x94\x8c]/linkhome/rech/gensak01/ugt86fu/.local/lib/python3.10/site-packages/torch/_inductor/config.py\x94ub\x8c\x08__spec__\x94\x8c\x11_frozen_importlib\x94\x8c\nModuleSpec\x94\x93\x94)\x81\x94}\x94(h\x0ch\x02\x8c\x06loader\x94h\n\x8c\x06origin\x94h\x0e\x8c\x0cloader_state\x94N\x8c\x1asubmodule_search_locations\x94N\x8c\r_set_fileattr\x94\x88\x8c\x07_cached\x94\x8cv/linkhome/rech/gensak01/ugt86fu/.local/lib/python3.10/site-packages/torch/_inductor/__pycache__/config.cpython-310.pyc\x94\x8c\r_initializing\x94\x89ub\x8c\x08__file__\x94h\x0e\x8c\n__cached__\x94h\x1b\x8c\x05debug\x94\x89\x8c\x10disable_progress\x94\x88\x8c\x10verbose_progress\x94\x89\x8c\x0bcpp_wrapper\x94\x89\x8c\x03dce\x94\x89\x8c\x14static_weight_shapes\x94\x88\x8c\x0csize_asserts\x94\x88\x8c\x10pick_loop_orders\x94\x88\x8c\x0finplace_buffers\x94\x88\x8c\x11benchmark_harness\x94\x88\x8c\x0fepilogue_fusion\x94\x89\x8c\x15epilogue_fusion_first\x94\x89\x8c\x0fpattern_matcher\x94\x88\x8c\nreordering\x94\x89\x8c\x0cmax_autotune\x94\x89\x8c\x17realize_reads_threshold\x94K\x04\x8c\x17realize_bytes_threshold\x94M\xd0\x07\x8c\x1brealize_acc_reads_threshold\x94K\x08\x8c\x0ffallback_random\x94\x89\x8c\x12implicit_fallbacks\x94\x88\x8c\rprefuse_nodes\x94\x88\x8c\x0btune_layout\x94\x89\x8c\x11aggressive_fusion\x94\x89\x8c\x0fmax_fusion_size\x94K@\x8c\x1bunroll_reductions_threshold\x94K\x08\x8c\x0ecomment_origin\x94\x89\x8c\tis_fbcode\x94h\x02h9\x93\x94\x8c\x0fcompile_threads\x94K\n\x8c\x13kernel_name_max_ops\x94K\n\x8c\x0finductor_import\x94\x8c\x0ftorch._inductor\x94\x8c\rshape_padding\x94\x89\x8c\x0epermute_fusion\x94\x89\x8c\x1aprofiler_mark_wrapper_call\x94\x89\x8c\x03cpp\x94}\x94(\x8c\n__module__\x94h\x02\x8c\x07threads\x94J\xff\xff\xff\xff\x8c\x0fdynamic_threads\x94\x89\x8c\x07simdlen\x94N\x8c\x0emin_chunk_size\x94M\x00\x10\x8c\x03cxx\x94N\x8c\x03g++\x94\x86\x94\x8c\x15enable_kernel_profile\x94\x89h\x03Nu\x8c\x06triton\x94}\x94(hDh\x02\x8c\ncudagraphs\x94\x88\x8c\x10debug_sync_graph\x94\x89\x8c\x11debug_sync_kernel\x94\x89\x8c\x0bconvolution\x94\x8c\x04aten\x94\x8c\x0edense_indexing\x94\x89\x8c\tmax_tiles\x94K\x02\x8c\x12autotune_pointwise\x94\x88\x8c tiling_prevents_pointwise_fusion\x94\x88\x8c tiling_prevents_reduction_fusion\x94\x88\x8c\x14ordered_kernel_names\x94\x89\x8c\x18descriptive_kernel_names\x94\x89h\x03Nu\x8c\x05trace\x94}\x94(hDh\x02\x8c\x07enabled\x94\x89\x8c\tdebug_log\x94\x88\x8c\x08info_log\x94\x89\x8c\x08fx_graph\x94\x88\x8c\x14fx_graph_transformed\x94\x88\x8c\rir_pre_fusion\x94\x88\x8c\x0eir_post_fusion\x94\x88\x8c\x0boutput_code\x94\x88\x8c\rgraph_diagram\x94\x89\x8c\x0fcompile_profile\x94\x89\x8c\nupload_tar\x94Nh\x03Nu\x8c\x15InductorConfigContext\x94}\x94(hDh\x02\x8c\x0f__annotations__\x94}\x94(\x8c\rstatic_memory\x94\x8c\x08builtins\x94\x8c\x04bool\x94\x93\x94\x8c\x0ematmul_padding\x94hoh-ho\x8c\x12triton_convolution\x94hm\x8c\x03str\x94\x93\x94\x8c\x17rematerialize_threshold\x94hm\x8c\x03int\x94\x93\x94\x8c\x1brematerialize_acc_threshold\x94hvu\x8c\x05_save\x94h\x02\x8c\x1bInductorConfigContext._save\x94\x93\x94\x8c\x06_apply\x94h\x02\x8c\x1cInductorConfigContext._apply\x94\x93\x94\x8c\x08__init__\x94h\x02\x8c\x1eInductorConfigContext.__init__\x94\x93\x94\x8c\t__enter__\x94h\x02\x8c\x1fInductorConfigContext.__enter__\x94\x93\x94\x8c\x08__exit__\x94h\x02\x8c\x1eInductorConfigContext.__exit__\x94\x93\x94h\x03Nu\x8c\x1cget_config_serialization_fns\x94\x8c\x1atorch._dynamo.config_utils\x94h\x87\x93\x94u.')


# REPLACEABLE COMMENT FOR TESTING PURPOSES


args = [((32, 3, 256, 256), (196608, 65536, 256, 1), torch.float32, 'cuda', False)]
args = [rand_strided(sh, st, dt, dev).requires_grad_(rg) for (sh, st, dt, dev, rg) in args]


from torch.nn import *
class Repro(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.self_encoder_conv1 = Conv2d(3, 64, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)).cuda()
        self.self_encoder_conv2 = Conv2d(64, 128, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)).cuda()



    def forward(self, x : torch.Tensor):
        self_encoder_conv1 = self.self_encoder_conv1(x);  x = None
        relu = torch.nn.functional.relu(self_encoder_conv1);  self_encoder_conv1 = None
        self_encoder_conv2 = self.self_encoder_conv2(relu);  relu = None
        relu_1 = torch.nn.functional.relu(self_encoder_conv2);  self_encoder_conv2 = None
        return (relu_1,)


mod = Repro()

# Setup debug minifier compiler
torch._dynamo.debug_utils.MINIFIER_SPAWNED = True
compiler_fn = BACKENDS["dynamo_minifier_backend"]

dynamo_minifier_backend = functools.partial(
    compiler_fn,
    compiler_name="inductor",
)
opt_mod = torch._dynamo.optimize(dynamo_minifier_backend)(mod)

with torch.cuda.amp.autocast(enabled=True):
    opt_mod(*args)

```

Please let me know if more information is needed.

### Versions

PyTorch version: 2.0.0.dev20230201+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.6 (Ootpa) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-10)
Clang version: Could not collect
CMake version: version 3.25.0
Libc version: glibc-2.28

Python version: 3.10.4 | packaged by conda-forge | (main, Mar 24 2022, 17:38:57) [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-372.36.1.el8_6.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: Tesla V100-SXM2-16GB
Nvidia driver version: 525.60.13
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              80
On-line CPU(s) list: 0-79
Thread(s) per core:  2
Core(s) per socket:  20
Socket(s):           2
NUMA node(s):        2
Vendor ID:           GenuineIntel
CPU family:          6
Model:               85
Model name:          Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
Stepping:            7
CPU MHz:             2500.000
CPU max MHz:         3900.0000
CPU min MHz:         1000.0000
BogoMIPS:            5000.00
Virtualization:      VT-x
L1d cache:           32K
L1i cache:           32K
L2 cache:            1024K
L3 cache:            28160K
NUMA node0 CPU(s):   0-19,40-59
NUMA node1 CPU(s):   20-39,60-79
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 cdp_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb intel_pt avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req pku ospke avx512_vnni md_clear flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] mypy==0.991
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.24.0rc2
[pip3] numpydoc==1.2.1
[pip3] pytorch-pfn-extras==0.6.2
[pip3] pytorch-triton==2.0.0+0d7e753227
[pip3] rotary-embedding-torch==0.2.1
[pip3] torch==2.0.0.dev20230201+cu117
[pip3] torchaudio==2.0.0.dev20230201+cu117
[pip3] torchmetrics==0.10.2
[pip3] torchtriton==2.0.0+0d7e753227
[pip3] torchvision==0.15.0.dev20230201+cu117
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] numpydoc                  1.2.1              pyhd8ed1ab_0    conda-forge

cc @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @H-Huang @kwen2501 @awgu @ezyang @msaroufim @wconstab @ngimel @bdhirsh @anijain2305 @soumith