torch._dynamo.exc.InternalTorchDynamoError: 'NoneType' object has no attribute 'guards'
Labels: triaged, oncall: pt2, module: dynamo

### 🐛 Describe the bug

Steps to reproduce:

pytorch version: 796ce672296c9ae8d7387b18403810aa2f1048a1
torchrec version: 005727ef06cdc808aa7d263ab4f2837938a77ed2

1. Get working torchrec install (you need fbgemm-gpu and torchrec, install by source; explicitly uninstall fbgemm-gpu-nightly and torchrec-nightly if they exist)
2. Patch torchrec with

```
diff --git a/examples/golden_training/train_dlrm.py b/examples/golden_training/train_dlrm.py
index 4004fbc..e22677d 100644
--- a/examples/golden_training/train_dlrm.py
+++ b/examples/golden_training/train_dlrm.py
@@ -151,8 +151,13 @@ def train(
             num_embeddings=num_embeddings,
         )
     )
-    for _ in tqdm(range(int(num_iterations)), mininterval=5.0):
-        train_pipeline.progress(train_iterator)
+
+    @torch.compile()
+    def f():
+        for _ in tqdm(range(int(num_iterations)), mininterval=5.0):
+            train_pipeline.progress(train_iterator)
+
+    f()
 
 
 if __name__ == "__main__":
```

3. In torchrec/examples/golden_training/ run `TORCH_LOGS=dynamo MASTER_ADDR=127.0.0.1 MASTER_PORT=29501 RANK=0 LOCAL_RANK=0 WORLD_SIZE=1 python train_dlrm.py`

Backtrace

```
Traceback (most recent call last):
  File "/data/users/ezyang/a/torchrec/examples/golden_training/train_dlrm.py", line 164, in <module>
    main()
  File "/data/users/ezyang/a/pytorch/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/data/users/ezyang/a/torchrec/examples/golden_training/train_dlrm.py", line 52, in main
    train()
  File "/data/users/ezyang/a/torchrec/examples/golden_training/train_dlrm.py", line 160, in train
    f()
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 333, in _fn
    return fn(*args, **kwargs)
  File "/data/users/ezyang/a/torchrec/examples/golden_training/train_dlrm.py", line 157, in f
    for _ in tqdm(range(int(num_iterations)), mininterval=5.0):
  File "/data/users/ezyang/a/torchrec/examples/golden_training/train_dlrm.py", line 158, in <resume in f>
    train_pipeline.progress(train_iterator)
  File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 987, in progress
    self._fill_pipeline(dataloader_iter)
  File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 979, in _fill_pipeline
    self._init_pipelined_modules(self._batch_i)
  File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 1028, in _init_pipelined_modules
    self._pipelined_modules, self._model = _rewrite_model(
  File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 860, in _rewrite_model
    arg_info_list, num_found = _get_node_args(node, feature_processor_nodes)
  File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 707, in _get_node_args
    pos_arg_info_list, num_found = _get_node_args_helper(
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/eval_frame.py", line 493, in catch_errors
    return callback(frame, cache_size, hooks, frame_state)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 626, in _convert_frame
    result = inner_convert(frame, cache_size, hooks, frame_state)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 133, in _fn
    return fn(*args, **kwargs)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 379, in _convert_frame_assert
    return _compile(
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 571, in _compile
    raise InternalTorchDynamoError(str(e)).with_traceback(
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 554, in _compile
    guarded_code = compile_inner(code, one_graph, hooks, transform)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/utils.py", line 181, in time_wrapper
    r = func(*args, **kwargs)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 476, in compile_inner
    out_code = transform_code_object(code, transform)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/bytecode_transformation.py", line 1028, in transform_code_object
    transformations(instructions, code_options)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/convert_frame.py", line 443, in transform
    tracer.run()
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/symbolic_convert.py", line 2074, in run
    super().run()
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/symbolic_convert.py", line 724, in run
    and self.step()
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/symbolic_convert.py", line 688, in step
    getattr(self, inst.opname)(inst)
  File "/data/users/ezyang/a/pytorch/torch/_dynamo/symbolic_convert.py", line 246, in inner
    self.output.guards.update(value.guards)
torch._dynamo.exc.InternalTorchDynamoError: 'NoneType' object has no attribute 'guards'
from user code:
   File "/data/users/ezyang/a/torchrec/torchrec/distributed/train_pipeline.py", line 656, in _get_node_args_helper
    and child_node in feature_processor_arguments
Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
```

Full dynamo debug log: https://gist.github.com/ezyang/deae63fd778eb63265111a3190561562

### Versions

main

cc @msaroufim @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov