RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation
Labels: triaged, bug, oncall: pt2

### 🐛 Describe the bug

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

### Error logs

[2023-01-17 16:14:06,580] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,584] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,587] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,617] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,638] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,638] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,671] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,679] torch._inductor.ir: [WARNING] DeviceCopy
[2023-01-17 16:14:06,726] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,736] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,743] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,759] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,778] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,787] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,820] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
[2023-01-17 16:14:06,831] torch._inductor.compile_fx: [WARNING] skipping cudagraphs due to multiple devices
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
/home/tiger/.local/lib/python3.7/site-packages/torch/cuda/graphs.py:82: UserWarning: The CUDA Graph is empty. This ususally means that the graph was attempted to be captured on wrong device or stream. (Triggered internally at ../aten/src/ATen/cuda/CUDAGraph.cpp:191.)
  super(CUDAGraph, self).capture_end()
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
        self, gradient, retain_graph, create_graph, inputs=inputsscaler.scale(loss).backward()

  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
        list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)main(args)

RuntimeError  File "fsdp_pretrain.py", line 318, in main
: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
text: Unable to write to output stream.
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
Traceback (most recent call last):
  File "fsdp_pretrain.py", line 423, in <module>
text: Unable to write to output stream.
text: Unable to write to output stream.
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    main(args)
  File "fsdp_pretrain.py", line 318, in main
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    scaler.scale(loss).backward()
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_tensor.py", line 489, in backward
    self, gradient, retain_graph, create_graph, inputs=inputs
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/__init__.py", line 199, in backward
    allow_unreachable=True, accumulate_grad=True)  # Calls into the C++ engine to run the backward pass
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/autograd/function.py", line 276, in apply
    return user_fn(self, *args)
  File "/home/tiger/.local/lib/python3.7/site-packages/torch/_functorch/aot_autograd.py", line 1848, in backward
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).
    list(ctx.symints) + list(ctx.saved_tensors) + list(contiguous_args)
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.HalfTensor [1024]] is at version 24; expected version 23 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).


cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh