Device "meta" and device "cuda:0" error

JasonW250: <p>I got a new gpu and installed the newest nightly build in order to use cuda 12.8.</p>
<p>On an older build of torch I never got this error, but with the same exact code on the newer build I get this:</p>
<p>E0423 07:47:56.453000 2930 torch/<em>subclasses/fake_tensor.py:2613] [1/1] failed while attempting to run meta for aten.set</em>.source_Storage<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] Traceback (most recent call last):<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 2609, in _dispatch_impl<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]     r = func(*args, **kwargs)<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]         ^^^^^^^^^^^^^^^^^^^^^<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_ops.py”, line 806, in <strong>call</strong><br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]     return self._op(*args, **kwargs)<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1]            ^^^^^^^^^^^^^^^^^^^^^^^^^<br>
E0423 07:47:56.453000 2930 torch/_subclasses/fake_tensor.py:2613] [1/1] RuntimeError: Attempted to set the storage of a tensor on device “meta” to a storage on different device “cuda:0”.  This is no longer allowed; the devices must match.<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] failed to eagerly compile backwards for dynamic, suppressing in case backwards not needed<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] Traceback (most recent call last):<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py”, line 1222, in aot_dispatch_autograd<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     copy.deepcopy(bw_module), placeholder_list<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     ^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y = copier(memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/fx/graph_module.py”, line 889, in <strong>deepcopy</strong><br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     fake_mod = _CodeOnlyModule(copy.deepcopy(self.<strong>dict</strong>, memo))<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 136, in deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y = copier(x, memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 221, in _deepcopy_dict<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                              ^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 136, in deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y = copier(x, memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 221, in _deepcopy_dict<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y[deepcopy(key, memo)] = deepcopy(value, memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                              ^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y = copier(memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_tensor.py”, line 172, in <strong>deepcopy</strong><br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     new_storage = self._typed_storage()._deepcopy(memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 1135, in _deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return self._new_wrapped_storage(copy.deepcopy(self._untyped_storage, memo))<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/usr/lib/python3.12/copy.py”, line 143, in deepcopy<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     y = copier(memo)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 239, in <strong>deepcopy</strong><br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     new_storage = self.clone()<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]                   ^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/storage.py”, line 253, in clone<br>
W0423 07:47:56.457000 2930 torch/_functorch/<em>aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return type(self)(self.nbytes(), device=self.device).copy</em>(self)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/utils/_stats.py”, line 27, in wrapper<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return fn(*args, **kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]            ^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1312, in <strong>torch_dispatch</strong><br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return self.dispatch(func, types, args, kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1950, in dispatch<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return self._cached_dispatch_impl(func, types, args, kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 1424, in _cached_dispatch_impl<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     output = self._dispatch_impl(func, types, args, kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_subclasses/fake_tensor.py”, line 2609, in _dispatch_impl<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     r = func(*args, **kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]         ^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]   File “/home/jason/myai2/lib/python3.12/site-packages/torch/_ops.py”, line 806, in <strong>call</strong><br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]     return self._op(*args, **kwargs)<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1]            ^^^^^^^^^^^^^^^^^^^^^^^^^<br>
W0423 07:47:56.457000 2930 torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py:1238] [1/1] RuntimeError: Attempted to set the storage of a tensor on device “meta” to a storage on different device “cuda:0”.  This is no longer allowed; the devices must match.</p>