NotImplementedError: could not run 'aten::as_strided'

username: mEm
cooked: <p>I am hitting this <code>NotImplementedError</code> with a custom MPI backend used with <code>DDP</code>.</p>
<pre><code class="lang-auto">[rank0]: Traceback (most recent call last):
[rank0]:   File ~/multinode_mpi.py", line 163, in &lt;module&gt;
[rank0]:     main(args)
[rank0]:   File ~/multinode_mpi.py", line 145, in main
[rank0]:     trainer.train(args.total_epochs)
[rank0]:   File ~/multinode_mpi.py", line 95, in train
[rank0]:     self._run_epoch(epoch)
[rank0]:   File ~/multinode_mpi.py", line 81, in _run_epoch
[rank0]:     self._run_batch(source, targets)
[rank0]:   File ~/multinode_mpi.py", line 68, in _run_batch
[rank0]:     loss.backward()
[rank0]:   File "~/.local/lib/python3.10/site-packages/torch/_tensor.py", line 650, in backward
[rank0]:     torch.autograd.backward(
[rank0]:   File "~/.local/lib/python3.10/site-packages/torch/autograd/__init__.py", line 354, in backward
[rank0]:     _engine_run_backward(
[rank0]:   File "~/.local/lib/python3.10/site-packages/torch/autograd/graph.py", line 825, in _engine_run_backward
[rank0]:     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
[rank0]: NotImplementedError: Could not run 'aten::as_strided' with arguments from the 'TESTING_ONLY_GenericWrapper' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::as_strided' is only available for these backends: [CPU, Meta, QuantizedCPU, BackendSelect, Python, FuncTorchDynamicLayerBackMode, Functionalize, Named, Conjugate, Negative, ZeroTensor, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradHIP, AutogradXLA, AutogradMPS, AutogradIPU, AutogradXPU, AutogradHPU, AutogradVE, AutogradLazy, AutogradMTIA, AutogradMAIA, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, AutogradMeta, AutogradNestedTensor, Tracer, AutocastCPU, AutocastMTIA, AutocastMAIA, AutocastXPU, AutocastMPS, AutocastCUDA, FuncTorchBatched, BatchedNestedTensor, FuncTorchVmapMode, Batched, VmapMode, FuncTorchGradWrapper, PythonTLSSnapshot, FuncTorchDynamicLayerFrontMode, PreDispatch, PythonDispatcher].

[rank0]: CPU: registered at ~/pytorch/build/aten/src/ATen/RegisterCPU_0.cpp:1830 [kernel]
[rank0]: Meta: registered at ~/pytorch/build/aten/src/ATen/RegisterMeta_0.cpp:2438 [kernel]
[rank0]: QuantizedCPU: registered at ~/pytorch/build/aten/src/ATen/RegisterQuantizedCPU_0.cpp:201 [kernel]
[rank0]: BackendSelect: fallthrough registered at ~/pytorch/aten/src/ATen/core/BackendSelectFallbackKernel.cpp:3 [backend fallback]
[rank0]: Python: registered at ~/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:194 [backend fallback]
[rank0]: FuncTorchDynamicLayerBackMode: registered at ~/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:479 [backend fallback]
[rank0]: Functionalize: registered at ~/pytorch/build/aten/src/ATen/RegisterFunctionalization_0.cpp:23380 [kernel]
[rank0]: Named: fallthrough registered at ~/pytorch/aten/src/ATen/core/NamedRegistrations.cpp:11 [kernel]
[rank0]: Conjugate: fallthrough registered at ~/pytorch/aten/src/ATen/ConjugateFallback.cpp:21 [kernel]
[rank0]: Negative: fallthrough registered at ~/pytorch/aten/src/ATen/native/NegateFallback.cpp:22 [kernel]
[rank0]: ZeroTensor: registered at ~/pytorch/build/aten/src/ATen/RegisterZeroTensor_0.cpp:146 [kernel]
[rank0]: ADInplaceOrView: registered at ~/pytorch/torch/csrc/autograd/generated/ADInplaceOrViewType_0.cpp:4960 [kernel]
[rank0]: AutogradOther: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradCPU: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradCUDA: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradHIP: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradXLA: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradMPS: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradIPU: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradXPU: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradHPU: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradVE: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradLazy: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradMTIA: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradMAIA: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradPrivateUse1: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradPrivateUse2: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradPrivateUse3: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradMeta: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: AutogradNestedTensor: registered at ~/pytorch/torch/csrc/autograd/generated/VariableType_0.cpp:18106 [autograd kernel]
[rank0]: Tracer: registered at ~/pytorch/torch/csrc/autograd/generated/TraceType_0.cpp:17190 [kernel]
[rank0]: AutocastCPU: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:322 [backend fallback]
[rank0]: AutocastMTIA: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:466 [backend fallback]
[rank0]: AutocastMAIA: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:504 [backend fallback]
[rank0]: AutocastXPU: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:542 [backend fallback]
[rank0]: AutocastMPS: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:209 [backend fallback]
[rank0]: AutocastCUDA: fallthrough registered at ~/pytorch/aten/src/ATen/autocast_mode.cpp:165 [backend fallback]
[rank0]: FuncTorchBatched: registered at ~/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:735 [kernel]
[rank0]: BatchedNestedTensor: registered at ~/pytorch/aten/src/ATen/functorch/LegacyBatchingRegistrations.cpp:758 [backend fallback]
[rank0]: FuncTorchVmapMode: fallthrough registered at ~/pytorch/aten/src/ATen/functorch/VmapModeRegistrations.cpp:27 [backend fallback]
[rank0]: Batched: registered at ~/pytorch/aten/src/ATen/LegacyBatchingRegistrations.cpp:1079 [kernel]
[rank0]: VmapMode: fallthrough registered at ~/pytorch/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]
[rank0]: FuncTorchGradWrapper: registered at ~/pytorch/aten/src/ATen/functorch/TensorWrapper.cpp:208 [backend fallback]
[rank0]: PythonTLSSnapshot: registered at ~/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:202 [backend fallback]
[rank0]: FuncTorchDynamicLayerFrontMode: registered at ~/pytorch/aten/src/ATen/functorch/DynamicLayer.cpp:475 [backend fallback]
[rank0]: PreDispatch: registered at ~/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:206 [backend fallback]
[rank0]: PythonDispatcher: registered at ~/pytorch/aten/src/ATen/core/PythonFallbackKernel.cpp:198 [backend fallback]
</code></pre>
<p>Now, I know this might seem too abstract.<br>
However, I’d love to know from Pytorch Devs where is the potential source of such error (logically). Why? Because with the same code (which I cannot share unfortunately), the same error does not show if I use the built-in MPI backend, while it does if I use the custom one.<br>
So, I guess the error relies in the distributed backends registration phase, in which possibly I am missing something to let the custom MPI backend being treated same as the built-in one.</p>
<p>Any help is highly appreciated.</p>

username: tianyu
cooked: <p>could you post an issue on <a href="https://github.com/pytorch/pytorch/issues" class="inline-onebox" rel="noopener nofollow ugc">GitHub · Where software is built</a></p>