cannot create weak reference to 'weakproxy' object in compile mode
Labels: triaged, module: inductor

### 🐛 Describe the bug

When I  use pytorch2.0 without compile, everything goes well. However, when I started the compile mode, a strange bug occurred.
```
[2023-02-11 15:53:18,520] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing validation_step
[2023-02-11 15:53:18,803] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing set_task
[2023-02-11 15:53:18,823] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing hparams
[2023-02-11 15:53:18,839] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in set_task>
[2023-02-11 15:53:18,888] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in validation_step>
[2023-02-11 15:53:19,130] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in validation_step>
[2023-02-11 15:53:19,359] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:19,552] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-02-11 15:53:19,580] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing img_embed
[2023-02-11 15:53:19,780] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:19,945] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:19,946] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:19,947] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:19,977] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:19,988] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:20,184] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:20,184] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:20,185] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:20,363] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 1
[2023-02-11 15:53:21,097] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 1
[2023-02-11 15:53:21,097] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:26,137] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:26,204] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __iter__
[2023-02-11 15:53:26,218] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:26,469] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:26,479] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:26,480] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:26,482] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:27,015] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 2
[2023-02-11 15:53:27,109] torch._inductor.graph: [INFO] Creating implicit fallback for:
  target: aten._scaled_dot_product_flash_attention.default
  args[0]: TensorBox(
    View(
      PermuteView(data=View(
        View(
          SliceView(
            StorageBox(
              Pointwise(
                'cuda',
                torch.float16,
                tmp0 = load(buf6, i3 + 768 * i0 + 2304 * i1)
                return tmp0
                ,
                ranges=[3, 197, 1, 768],
                origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
              )
            ),
            size=[1, 197, 1, 768],
            reindex=lambda i0, i1, i2, i3: [i0, i1, i2, i3],
            origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, select, mul_1, view_1, rsqrt, convert_element_type_2}
          ),
          size=(197, 1, 768),
          reindex=lambda i0, i1, i2: [0, i0, 0, i2],
          origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, select, mul_1, view_1, rsqrt, convert_element_type_2}
        ),
        size=(197, 12, 64),
        reindex=lambda i0, i1, i2: [i0, 0, 64*i1 + i2],
        origins={convert_element_type, convert_element_type_1, view_3, arg6_1, clone, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, select, mul_1, view_1, rsqrt, convert_element_type_2}
      ), dims=[1, 0, 2]),
      size=(1, 12, 197, 64),
      reindex=lambda i0, i1, i2, i3: [i1, i2, i3],
      origins={permute_2, convert_element_type, convert_element_type_1, view_3, arg6_1, clone, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view_6, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, select, mul_1, view_1, rsqrt, convert_element_type_2}
    )
  )
  args[1]: TensorBox(
    View(
      PermuteView(data=View(
        View(
          SliceView(
            StorageBox(
              Pointwise(
                'cuda',
                torch.float16,
                tmp0 = load(buf6, i3 + 768 * i0 + 2304 * i1)
                return tmp0
                ,
                ranges=[3, 197, 1, 768],
                origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
              )
            ),
            size=[1, 197, 1, 768],
            reindex=lambda i0, i1, i2, i3: [i0 + 1, i1, i2, i3],
            origins={convert_element_type, convert_element_type_1, clone, arg6_1, select_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
          ),
          size=(197, 1, 768),
          reindex=lambda i0, i1, i2: [0, i0, 0, i2],
          origins={convert_element_type, convert_element_type_1, clone, arg6_1, select_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
        ),
        size=(197, 12, 64),
        reindex=lambda i0, i1, i2: [i0, 0, 64*i1 + i2],
        origins={convert_element_type, convert_element_type_1, clone, arg6_1, select_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view_4, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
      ), dims=[1, 0, 2]),
      size=(1, 12, 197, 64),
      reindex=lambda i0, i1, i2, i3: [i1, i2, i3],
      origins={view_7, convert_element_type, convert_element_type_1, clone, arg6_1, select_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, permute_3, add, var_mean, sub, arg8_1, view_4, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
    )
  )
  args[2]: TensorBox(
    View(
      PermuteView(data=View(
        View(
          SliceView(
            StorageBox(
              Pointwise(
                'cuda',
                torch.float16,
                tmp0 = load(buf6, i3 + 768 * i0 + 2304 * i1)
                return tmp0
                ,
                ranges=[3, 197, 1, 768],
                origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
              )
            ),
            size=[1, 197, 1, 768],
            reindex=lambda i0, i1, i2, i3: [i0 + 2, i1, i2, i3],
            origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, select_2, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
          ),
          size=(197, 1, 768),
          reindex=lambda i0, i1, i2: [0, i0, 0, i2],
          origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, arg7_1, addmm, mul, var_mean, select_2, add, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2}
        ),
        size=(197, 12, 64),
        reindex=lambda i0, i1, i2: [i0, 0, 64*i1 + i2],
        origins={convert_element_type, convert_element_type_1, clone, arg6_1, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, select_2, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2, view_5}
      ), dims=[1, 0, 2]),
      size=(1, 12, 197, 64),
      reindex=lambda i0, i1, i2, i3: [i1, i2, i3],
      origins={convert_element_type, convert_element_type_1, permute_4, arg6_1, clone, arg1_1, squeeze, arg0_1, addmm, arg7_1, mul, var_mean, add, select_2, view_8, sub, arg8_1, view, add_1, permute, unsqueeze, convert_element_type_3, permute_1, view_2, view_1, mul_1, rsqrt, convert_element_type_2, view_5}
    )
  )
[2023-02-11 15:53:27,133] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:27,786] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 2
[2023-02-11 15:53:28,015] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 3
[2023-02-11 15:53:28,170] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 3
[2023-02-11 15:53:28,197] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:28,224] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:28,456] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:28,465] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:28,466] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:28,467] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:28,977] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 4
[2023-02-11 15:53:29,055] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:29,406] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 4
[2023-02-11 15:53:29,629] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 5
[2023-02-11 15:53:29,684] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 5
[2023-02-11 15:53:29,711] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:29,727] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:29,955] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:29,964] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:29,964] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:29,965] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:30,471] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 6
[2023-02-11 15:53:30,544] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:30,873] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 6
[2023-02-11 15:53:31,079] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 7
[2023-02-11 15:53:31,135] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 7
[2023-02-11 15:53:31,160] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:31,178] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:31,404] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:31,412] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:31,413] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:31,413] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:31,924] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 8
[2023-02-11 15:53:32,001] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:32,805] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 8
[2023-02-11 15:53:33,029] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 9
[2023-02-11 15:53:33,085] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 9
[2023-02-11 15:53:33,113] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:33,130] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:33,372] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:33,382] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:33,383] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:33,383] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:33,935] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 10
[2023-02-11 15:53:34,050] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:34,439] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 10
[2023-02-11 15:53:34,658] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 11
[2023-02-11 15:53:34,713] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 11
[2023-02-11 15:53:34,741] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:34,757] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:34,984] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:34,993] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:34,993] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:34,994] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:35,496] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 12
[2023-02-11 15:53:35,570] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:35,918] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 12
[2023-02-11 15:53:36,131] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 13
[2023-02-11 15:53:36,185] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 13
[2023-02-11 15:53:36,211] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:36,227] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:36,448] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:36,456] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:36,460] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:36,461] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:36,987] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 14
[2023-02-11 15:53:37,064] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:37,400] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 14
[2023-02-11 15:53:37,611] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 15
[2023-02-11 15:53:37,665] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 15
[2023-02-11 15:53:37,694] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:37,711] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:37,943] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:37,952] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:37,953] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:37,953] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:38,503] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 16
[2023-02-11 15:53:38,596] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:38,956] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 16
[2023-02-11 15:53:39,171] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 17
[2023-02-11 15:53:39,227] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 17
[2023-02-11 15:53:39,254] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:39,270] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:39,503] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:39,511] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:39,512] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:39,513] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:40,028] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 18
[2023-02-11 15:53:40,103] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:40,433] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 18
[2023-02-11 15:53:40,654] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 19
[2023-02-11 15:53:40,710] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 19
[2023-02-11 15:53:40,737] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:40,755] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:41,031] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:41,039] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:41,040] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:41,041] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:41,559] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 20
[2023-02-11 15:53:41,641] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:42,077] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 20
[2023-02-11 15:53:42,298] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 21
[2023-02-11 15:53:42,353] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 21
[2023-02-11 15:53:42,380] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:42,397] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:53:42,659] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing forward (RETURN_VALUE)
[2023-02-11 15:53:42,668] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:42,669] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:42,669] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:43,235] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 22
[2023-02-11 15:53:43,313] torch._inductor.graph: [INFO] Using FallbackKernel: torch.ops.aten._scaled_dot_product_flash_attention.default
[2023-02-11 15:53:43,671] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 22
[2023-02-11 15:53:43,885] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 23
[2023-02-11 15:53:43,942] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 23
[2023-02-11 15:53:43,970] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:43,986] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in forward>
[2023-02-11 15:53:44,022] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <graph break in forward> (RETURN_VALUE)
[2023-02-11 15:53:44,026] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:44,026] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:44,027] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:44,111] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 24
[2023-02-11 15:53:44,328] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 24
[2023-02-11 15:53:44,329] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:44,343] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in img_embed>
[2023-02-11 15:53:45,086] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing <graph break in img_embed> (RETURN_VALUE)
[2023-02-11 15:53:45,112] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:53:45,112] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:53:45,113] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:53:45,287] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 25
[2023-02-11 15:53:45,517] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 25
[2023-02-11 15:53:45,518] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:53:45,557] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing compute_mlm
[2023-02-11 15:53:51,838] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing compute_mlm
[2023-02-11 15:54:08,780] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:54:08,807] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:54:08,808] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:54:09,651] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 26
[2023-02-11 15:54:09,668] torch._inductor.graph: [INFO] Using FallbackKernel: aten.cumsum
[2023-02-11 15:54:10,107] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 26
[2023-02-11 15:54:11,043] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 27
[2023-02-11 15:54:12,051] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 27
[2023-02-11 15:54:13,103] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 28
[2023-02-11 15:54:13,765] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 28
[2023-02-11 15:54:14,803] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 29
[2023-02-11 15:54:15,509] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 29
[2023-02-11 15:54:16,533] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 30
[2023-02-11 15:54:17,213] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 30
[2023-02-11 15:54:18,206] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 31
[2023-02-11 15:54:18,868] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 31
[2023-02-11 15:54:19,970] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 32
[2023-02-11 15:54:20,916] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 32
[2023-02-11 15:54:22,055] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 33
[2023-02-11 15:54:22,737] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 33
[2023-02-11 15:54:23,835] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 34
[2023-02-11 15:54:24,599] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 34
[2023-02-11 15:54:25,682] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 35
[2023-02-11 15:54:26,479] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 35
[2023-02-11 15:54:27,546] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 36
[2023-02-11 15:54:28,239] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 36
[2023-02-11 15:54:29,346] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 37
[2023-02-11 15:54:30,090] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 37
[2023-02-11 15:54:31,184] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 38
[2023-02-11 15:54:31,997] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 38
[2023-02-11 15:54:33,528] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 39
[2023-02-11 15:54:34,738] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 39
[2023-02-11 15:54:36,347] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 40
[2023-02-11 15:54:37,657] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 40
[2023-02-11 15:54:38,877] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 41
[2023-02-11 15:54:39,850] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 41
[2023-02-11 15:54:41,383] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 42
[2023-02-11 15:54:42,426] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 42
[2023-02-11 15:54:43,640] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 43
[2023-02-11 15:54:44,678] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 43
[2023-02-11 15:54:46,243] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 44
[2023-02-11 15:54:47,361] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 44
[2023-02-11 15:54:48,661] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 45
[2023-02-11 15:54:49,416] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 45
[2023-02-11 15:54:50,510] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 46
[2023-02-11 15:54:51,304] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 46
[2023-02-11 15:54:52,734] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 47
[2023-02-11 15:54:53,787] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 47
[2023-02-11 15:54:55,427] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 48
[2023-02-11 15:54:56,426] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 48
[2023-02-11 15:54:57,527] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 49
[2023-02-11 15:54:58,262] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 49
[2023-02-11 15:54:59,879] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 50
[2023-02-11 15:55:00,885] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 50
[2023-02-11 15:55:02,138] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 51
[2023-02-11 15:55:02,887] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 51
[2023-02-11 15:55:04,413] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 52
[2023-02-11 15:55:05,501] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 52
[2023-02-11 15:55:06,982] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 53
[2023-02-11 15:55:08,648] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 53
[2023-02-11 15:55:09,754] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 54
[2023-02-11 15:55:10,600] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 54
[2023-02-11 15:55:10,848] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 55
[2023-02-11 15:55:10,959] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 55
[2023-02-11 15:55:10,993] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:11,219] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __getitem__
[2023-02-11 15:55:11,236] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing _get_abs_string_index
[2023-02-11 15:55:11,246] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing __len__
[2023-02-11 15:55:11,288] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:11,368] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:11,431] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:11,489] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:11,533] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:11,534] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:11,534] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:11,615] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 56
[2023-02-11 15:55:11,908] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 56
[2023-02-11 15:55:11,909] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:11,963] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:55:11,993] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing _forward_full_state_update
[2023-02-11 15:55:12,022] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing wrapped_func
[2023-02-11 15:55:12,049] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in wrapped_func>
[2023-02-11 15:55:12,077] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in wrapped_func>
[2023-02-11 15:55:12,093] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing update
[2023-02-11 15:55:12,111] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:12,111] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:12,111] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:12,147] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 57
[2023-02-11 15:55:12,194] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 57
[2023-02-11 15:55:12,195] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:12,212] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:12,220] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:12,220] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:12,221] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:12,246] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 58
[2023-02-11 15:55:12,295] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 58
[2023-02-11 15:55:12,296] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:12,311] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:12,320] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,351] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,378] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,408] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,456] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,486] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing reset
[2023-02-11 15:55:12,504] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in reset>
[2023-02-11 15:55:12,523] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in reset>
[2023-02-11 15:55:12,542] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in reset>
[2023-02-11 15:55:12,572] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,608] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,644] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing wrapped_func
[2023-02-11 15:55:12,670] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in wrapped_func>
[2023-02-11 15:55:12,679] torch._dynamo.convert_frame: [INFO] converting frame raised unsupported, leaving it unconverted
[2023-02-11 15:55:12,686] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing sync
[2023-02-11 15:55:12,711] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in sync>
[2023-02-11 15:55:12,722] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing jit_distributed_available
[2023-02-11 15:55:12,732] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing compute
[2023-02-11 15:55:12,738] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing compute (RETURN_VALUE)
[2023-02-11 15:55:12,741] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:12,741] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:12,742] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:12,775] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 59
[2023-02-11 15:55:12,825] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 59
[2023-02-11 15:55:12,826] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:12,837] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing _squeeze_if_scalar
[2023-02-11 15:55:12,850] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing _squeeze_if_scalar (RETURN_VALUE)
[2023-02-11 15:55:12,853] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:12,853] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:12,854] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:12,875] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 60
[2023-02-11 15:55:12,878] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 60
[2023-02-11 15:55:12,879] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:12,887] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing unsync
[2023-02-11 15:55:12,901] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:12,913] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in forward>
[2023-02-11 15:55:12,929] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in forward>
[2023-02-11 15:55:12,950] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:12,979] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:13,048] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing forward
[2023-02-11 15:55:13,075] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing _forward_full_state_update
[2023-02-11 15:55:13,094] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing wrapped_func
[2023-02-11 15:55:13,109] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in wrapped_func>
[2023-02-11 15:55:13,125] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in wrapped_func>
[2023-02-11 15:55:13,141] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing update
[2023-02-11 15:55:13,168] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:13,168] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:13,169] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:13,217] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 61
[2023-02-11 15:55:13,306] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 61
[2023-02-11 15:55:13,307] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:13,328] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:13,336] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:13,336] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:13,337] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:13,368] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 62
[2023-02-11 15:55:13,397] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 62
[2023-02-11 15:55:13,398] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:13,422] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:13,440] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:13,441] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:13,441] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:13,482] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 63
[2023-02-11 15:55:13,964] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 63
[2023-02-11 15:55:13,965] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:14,004] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:14,018] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:14,018] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:14,019] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:14,051] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 64
[2023-02-11 15:55:14,155] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 64
[2023-02-11 15:55:14,156] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:14,175] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in update>
[2023-02-11 15:55:14,186] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,206] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,224] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,239] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,273] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,288] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing reset
[2023-02-11 15:55:14,298] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in reset>
[2023-02-11 15:55:14,307] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in reset>
[2023-02-11 15:55:14,323] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,347] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in _forward_full_state_update>
[2023-02-11 15:55:14,367] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing wrapped_func
[2023-02-11 15:55:14,385] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing sync
[2023-02-11 15:55:14,398] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing compute
[2023-02-11 15:55:14,405] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo done tracing compute (RETURN_VALUE)
[2023-02-11 15:55:14,408] torch._dynamo.output_graph: [INFO] Step 2: calling compiler function compile_fn
[2023-02-11 15:55:14,408] torch._dynamo.backends.distributed: [INFO] DDPOptimizer used bucket cap 26214400 and produced the following buckets:
[2023-02-11 15:55:14,409] torch._dynamo.backends.distributed: [INFO] Please `pip install tabulate` in order to pretty-print ddp bucket sizes
[2023-02-11 15:55:14,435] torch._inductor.compile_fx: [INFO] Step 3: torchinductor compiling FORWARDS graph 65
[2023-02-11 15:55:14,451] torch._inductor.compile_fx: [INFO] Step 3: torchinductor done compiling FORWARDS graph 65
[2023-02-11 15:55:14,452] torch._dynamo.output_graph: [INFO] Step 2: done compiler function compile_fn
[2023-02-11 15:55:14,463] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in forward>
[2023-02-11 15:55:14,485] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in compute_mlm>
[2023-02-11 15:55:14,576] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing log
[2023-02-11 15:55:14,661] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing apply_to_collection
[2023-02-11 15:55:14,711] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing is_dataclass_instance
[2023-02-11 15:55:14,724] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in is_dataclass_instance>
[2023-02-11 15:55:14,738] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in apply_to_collection>
[2023-02-11 15:55:14,753] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in log>
[2023-02-11 15:55:14,836] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing apply_to_collection
[2023-02-11 15:55:14,867] torch._dynamo.symbolic_convert: [INFO] Step 1: torchdynamo start tracing <graph break in log>
ERROR - METER - Failed after 0:02:55!
Traceback (most recent calls WITHOUT Sacred internals):
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 364, in _compile
    check_fn = CheckFunctionManager(
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/guards.py", line 549, in __init__
    guard.create(local_builder, global_builder)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_guards.py", line 163, in create
    return self.create_fn(self.source.select(local_builder, global_builder), self)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/guards.py", line 172, in TYPE_MATCH
    self._produce_guard_code(guard, [code])
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/guards.py", line 473, in _produce_guard_code
    obj_ref = weakref.ref(guarded_object)
TypeError: cannot create weak reference to 'weakproxy' object

Set torch._dynamo.config.verbose=True for more information


You can suppress this exception and fall back to eager by setting:
    torch._dynamo.config.suppress_errors = True


The above exception was the direct cause of the following exception:

Traceback (most recent calls WITHOUT Sacred internals):
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/run.py", line 83, in main
    trainer.fit(model, datamodule=dm)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 608, in fit
    call._call_and_handle_interrupt(
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/call.py", line 36, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 88, in launch
    return function(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 650, in _fit_impl
    self._run(model, ckpt_path=self.ckpt_path)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1112, in _run
    results = self._run_stage()
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1191, in _run_stage
    self._run_train()
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1204, in _run_train
    self._run_sanity_check()
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1276, in _run_sanity_check
    val_loop.run()
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 152, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/loops/loop.py", line 199, in run
    self.advance(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 137, in advance
    output = self._evaluation_step(**kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 234, in _evaluation_step
    output = self.trainer._call_strategy_hook(hook_name, *kwargs.values())
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1494, in _call_strategy_hook
    output = fn(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 359, in validation_step
    return self.model(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1159, in forward
    output = self._run_ddp_forward(*inputs, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 1113, in _run_ddp_forward
    return module_to_run(*inputs[0], **kwargs[0])  # type: ignore[index]
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 110, in forward
    return self._forward_module.validation_step(*inputs, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/meter_module.py", line 420, in validation_step
    meter_utils.set_task(self)
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/meter_module.py", line 421, in <graph break in validation_step>
    output = self(batch)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/meter_module.py", line 361, in forward
    ret.update(objectives.compute_mlm(self, batch, ret['dual_image_feats']))
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 17, in compute_mlm
    def compute_mlm(pl_module, batch, dual_image_feats=None):
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 43, in <graph break in compute_mlm>
    mlm_logits.view(-1, pl_module.hparams.config["vocab_size"]),
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 43, in <graph break in compute_mlm>
    mlm_logits.view(-1, pl_module.hparams.config["vocab_size"]),
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 56, in <graph break in compute_mlm>
    loss = getattr(pl_module, f"{phase}_mlm_loss")(ret["mlm_loss"])
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 57, in <graph break in compute_mlm>
    acc = getattr(pl_module, f"{phase}_mlm_accuracy")(
  File "/mnt/vepfs/workspace/zhenyang/VL_model/model/meter/modules/objectives.py", line 57, in <graph break in compute_mlm>
    acc = getattr(pl_module, f"{phase}_mlm_accuracy")(
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 409, in log
    apply_to_collection(value, dict, self.__check_not_nested, name)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/pytorch_lightning/core/module.py", line 410, in <graph break in log>
    apply_to_collection(
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/eval_frame.py", line 327, in catch_errors
    return hijacked_callback(frame, cache_size, hooks)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 404, in _convert_frame
    result = inner_convert(frame, cache_size, hooks)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 104, in _fn
    return fn(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 262, in _convert_frame_assert
    return _compile(
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/utils.py", line 163, in time_wrapper
    r = func(*args, **kwargs)
  File "/mnt/vepfs/miniconda3/envs/nlp2/lib/python3.8/site-packages/torch/_dynamo/convert_frame.py", line 394, in _compile
    raise InternalTorchDynamoError() from e
torch._dynamo.exc.InternalTorchDynamoError

node30:1755820:1756503 [0] NCCL INFO [Service thread] Connection closed by localRank 0
node30:1755820:1755820 [0] NCCL INFO comm 0xa4b24900 rank 0 nranks 1 cudaDev 0 busId 69010 - Abort COMPLETE

Process finished with exit code 1
```


### Versions

NVIDIA A100-SXM-80GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A100-SXM-80GB GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/

  warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))
PyTorch version: 1.12.1+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.9.12 (main, Apr  5 2022, 06:56:58)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-104-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: 
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM-80GB
GPU 1: NVIDIA A100-SXM-80GB
GPU 2: NVIDIA A100-SXM-80GB
GPU 3: NVIDIA A100-SXM-80GB
GPU 4: NVIDIA A100-SXM-80GB
GPU 5: NVIDIA A100-SXM-80GB
GPU 6: NVIDIA A100-SXM-80GB
GPU 7: NVIDIA A100-SXM-80GB

Nvidia driver version: 470.57.02
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 57 bits virtual
CPU(s):                          112
On-line CPU(s) list:             0-111
Thread(s) per core:              2
Core(s) per socket:              28
Socket(s):                       2
NUMA node(s):                    2
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           106
Model name:                      Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
Stepping:                        6
CPU MHz:                         2300.056
BogoMIPS:                        4600.11
Hypervisor vendor:               KVM
Virtualization type:             full
L1d cache:                       2.6 MiB
L1i cache:                       1.8 MiB
L2 cache:                        70 MiB
L3 cache:                        54 MiB
NUMA node0 CPU(s):               0-55
NUMA node1 CPU(s):               56-111
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid md_clear arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==1.12.1
[pip3] torch-cluster==1.6.0
[pip3] torch-scatter==2.0.9
[pip3] torchdrug==0.2.0.post1
[pip3] torchtyping==0.1.4
[pip3] torchvision==0.13.1
[conda] blas                      1.0                         mkl    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl                       2021.4.0           h06a4308_640    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl-service               2.4.0            py39h7f8727e_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_fft                   1.3.1            py39hd3c417c_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl_random                1.2.2            py39h51133e4_0    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] numpy                     1.24.1                   pypi_0    pypi
[conda] numpy-base                1.21.5           py39hf524024_2    https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] torch                     1.12.1                   pypi_0    pypi
[conda] torch-cluster             1.6.0                    pypi_0    pypi
[conda] torch-scatter             2.0.9                    pypi_0    pypi
[conda] torchdrug                 0.2.0.post1              pypi_0    pypi
[conda] torchtyping               0.1.4                    pypi_0    pypi
[conda] torchvision               0.13.1                   pypi_0    pypi


cc @mlazos @soumith @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @desertfire