Using pytorch 2.0.0a0+gitfd3a726，Parallel training with 4 cards 4090 cannot be performed on AMD 5975WX， stuck at the beginning

MI_N: <p>I use a CPU of AMD 5975WX, and four 4090 graphics cards. cuda version is cuda12, pytoch version is 2.0`P</p>
<pre><code class="lang-auto">sudo gdb python 23557
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later &lt;http://gnu.org/licenses/gpl.html&gt;
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
&lt;https://www.gnu.org/software/gdb/bugs/&gt;.
Find the GDB manual and other documentation resources online at:
    &lt;http://www.gnu.org/software/gdb/documentation/&gt;.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
python: No such file or directory.
Attaching to process 23557
(gdb) bt
#0  0x00007ff8cda80680 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#1  0x00007ff8cddd71ef in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#2  0x00007ff8cddd9aef in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#3  0x00007ff8cda8ef9f in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#4  0x00007ff8cdcabd7c in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#5  0x00007ff8cddd01c9 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#6  0x00007ff8cda4f833 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#7  0x00007ff8cda4fd41 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#8  0x00007ff8cda508c8 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#9  0x00007ff8cdc1b381 in ?? () from /lib/x86_64-linux-gnu/libcuda.so.1
#10 0x00007ff874a42f49 in ?? ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libcudart-e409450e.so.11.0
#11 0x00007ff874a16e2d in ?? ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libcudart-e409450e.so.11.0
#12 0x00007ff874a67875 in cudaMemcpyAsync ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libcudart-e409450e.so.11.0
#13 0x00007ff87635ef28 in at::native::copy_kernel_cuda(at::TensorIterator&amp;, bool) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cuda_cu.so
#14 0x00007ff8a2e3bff9 in at::native::copy_impl(at::Tensor&amp;, at::Tensor const&amp;, bool) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007ff8a2e3d472 in at::native::copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007ff8a39d443f in at::_ops::copy_::call(at::Tensor&amp;, at::Tensor const&amp;, bool) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007ff8a312cf48 in at::native::_to_copy(at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007ff8a3d1887b in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;), &amp;at::(anonymous namespace)::(anonymous namespace)::wrapper___to_copy&gt;, at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt; &gt; &gt;, at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;)&gt;::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#19 0x00007ff8a3566165 in at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;)
    () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#20 0x00007ff8a3b5ddd3 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional----Type &lt;RET&gt; for more,--Type &lt;RET&gt;--Type &lt;RET&gt; f--Type &lt;RET&gt; f--Type &lt;RET&gt; for--Type &lt;RET&gt; for--Type &lt;RET&gt; for more, q to quit, --Type &lt;RET&gt; for more, q to quit, c to continu--Type &lt;RET&gt;--Type &lt;RET&gt;--Type &lt;RET&gt; for--Type &lt;--Type &lt;RET&gt; for more, q to quit, c to continue without paging--
&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;), &amp;at::(anonymous namespace)::_to_copy&gt;, at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt; &gt; &gt;, at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;)&gt;::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#21 0x00007ff8a3566165 in at::_ops::_to_copy::redispatch(c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#22 0x00007ff8a4f20b1b in torch::autograd::VariableType::(anonymous namespace)::_to_copy(c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#23 0x00007ff8a4f20f8e in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;at::Tensor (c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;), &amp;torch::autograd::VariableType::(anonymous namespace)::_to_copy&gt;, at::Tensor, c10::guts::typelist::typelist&lt;c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt; &gt; &gt;, at::Tensor (c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;)&gt;::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#24 0x00007ff8a35e66c9 in at::_ops::_to_copy::call(at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, c10::optional&lt;c10::MemoryFormat&gt;) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#25 0x00007ff8a312546b in at::native::to(at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#26 0x00007ff8a3ed6c81 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;), &amp;at::(anonymous namespace)::(anonymous namespace)::wrapper_dtype_layout_to&gt;, at::Tensor, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt; &gt; &gt;, at::Tensor (at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;)&gt;::call(c10::OperatorKernel*, c10::DispatchKeySet, at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#27 0x00007ff8a374d81e in at::_ops::to_dtype_layout::call(at::Tensor const&amp;, c10::optional&lt;c10::ScalarType&gt;, c10::optional&lt;c10::Layout&gt;, c10::optional&lt;c10::Device&gt;, c10::optional&lt;bool&gt;, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;) ()
   from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_cpu.so
#28 0x00007ff8cca7a318 in torch::autograd::dispatch_to(at::Tensor const&amp;, c10::Device, bool, bool, c10::optional&lt;c10::MemoryFormat&gt;) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#29 0x00007ff8cca7b780 in torch::autograd::THPVariable_cuda(_object*, _object*, _object*) () from /home/ubuntu/miniconda3/lib/python3.9/site-packages/torch/lib/libtorch_python.so
#30 0x0000555b32cc8b9c in method_vectorcall_VARARGS_KEYWORDS (func=&lt;optimized out&gt;, args=&lt;optimized out&gt;, nargsf=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;)
    at /opt/conda/conda-bld/python-split_1649141344976/work/Objects/descrobject.c:348
#31 0x0000555b32c0872f in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=&lt;optimized out&gt;, args=0x555b34d983e0, callable=0x7ff8e3fc1db0, tstate=&lt;optimized out&gt;)
    at /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:118
#32 PyObject_Vectorcall () at /opt/conda/conda-bld/python-split_1649141344976/work/Include/cpython/abstract.h:127
#33 call_function (kwnames=0x0, oparg=&lt;optimized out&gt;, pp_stack=&lt;synthetic pointer&gt;, tstate=0x555b34d3dc00) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:5077
#34 _PyEval_EvalFrameDefault (tstate=&lt;optimized out&gt;, f=0x555b34d98260, throwflag=&lt;optimized out&gt;) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:3506
#35 0x0000555b32c9f663 in _PyEval_EvalFrame () at /opt/conda/conda-bld/python-split_1649141344976/work/Include/internal/pycore_ceval.h:40
#36 _PyEval_EvalCode (tstate=&lt;optimized out&gt;, _co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kwnames=0x0, kwargs=0x0, kwcount=&lt;optimized out&gt;, kwstep=2, defs=0x0,
    defcount=0, kwdefs=0x0, closure=0x0, name=&lt;optimized out&gt;, qualname=0x0) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4329
#37 0x0000555b32d4c45c in _PyEval_EvalCodeWithName (qualname=0x0, name=0x0, closure=0x0, kwdefs=0x0, defcount=0, defs=0x0, kwstep=2, kwcount=0, kwargs=&lt;optimized out&gt;, kwnames=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;,
    args=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, _co=&lt;optimized out&gt;) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4361
#38 PyEval_EvalCodeEx (_co=&lt;optimized out&gt;, globals=&lt;optimized out&gt;, locals=&lt;optimized out&gt;, args=&lt;optimized out&gt;, argcount=&lt;optimized out&gt;, kws=&lt;optimized out&gt;, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0, closure=0x0)
    at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:4377
#39 0x0000555b32ca045b in PyEval_EvalCode (co=co@entry=0x7ff98e9c6660, globals=globals@entry=0x7ff98ea68d40, locals=locals@entry=0x7ff98ea68d40) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/ceval.c:828
#40 0x0000555b32d4c50b in run_eval_code_obj (tstate=0x555b34d3dc00, co=0x7ff98e9c6660, globals=0x7ff98ea68d40, locals=0x7ff98ea68d40) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1221
#41 0x0000555b32d7cf75 in run_mod (mod=&lt;optimized out&gt;, filename=&lt;optimized out&gt;, globals=0x7ff98ea68d40, locals=0x7ff98ea68d40, flags=&lt;optimized out&gt;, arena=&lt;optimized out&gt;)
    at /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1242
#42 0x0000555b32c1d987 in pyrun_file (fp=0x555b34d39340, filename=0x7ff98e9a4750, start=&lt;optimized out&gt;, globals=0x7ff98ea68d40, locals=0x7ff98ea68d40, closeit=1, flags=0x7ffc08274e28)
    at /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:1140
#43 0x0000555b32d82a2f in pyrun_simple_file (flags=0x7ffc08274e28, closeit=1, filename=0x7ff98e9a4750, fp=0x555b34d39340) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:450
#44 PyRun_SimpleFileExFlags (fp=0x555b34d39340, filename=&lt;optimized out&gt;, closeit=1, flags=0x7ffc08274e28) at /opt/conda/conda-bld/python-split_1649141344976/work/Python/pythonrun.c:483
#45 0x0000555b32d8310b in pymain_run_file (cf=0x7ffc08274e28, config=0x555b34d3c3b0) at /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:379
#46 pymain_run_python (exitcode=0x7ffc08274e20) at /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:604
#47 Py_RunMain () at /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:683
#48 0x0000555b32d83309 in Py_BytesMain (argc=&lt;optimized out&gt;, argv=&lt;optimized out&gt;) at /opt/conda/conda-bld/python-split_1649141344976/work/Modules/main.c:1129
#49 0x00007ff98f9a2d90 in __libc_start_call_main (main=main@entry=0x555b32c2a4a0 &lt;main&gt;, argc=argc@entry=2, argv=argv@entry=0x7ffc08275048) at ../sysdeps/nptl/libc_start_call_main.h:58
#50 0x00007ff98f9a2e40 in __libc_start_main_impl (main=0x555b32c2a4a0 &lt;main&gt;, argc=2, argv=0x7ffc08275048, init=&lt;optimized out&gt;, fini=&lt;optimized out&gt;, rtld_fini=&lt;optimized out&gt;, stack_end=0x7ffc08275038) at ../csu/libc-start.c:392
#51 0x0000555b32d0a0a0 in _start ()
(gdb)
</code></pre>
<p>See attachment for cuda log，My code is also in the attachment</p>
<pre><code class="lang-auto">import numpy as np
import os
import time
import PIL
import torch
from torch import Tensor
from torch.utils.data import Dataset, DataLoader
import torchvision
from torchvision import transforms
from torchvision.models.resnet import ResNet, BasicBlock

batch_size = 32
max_number_of_epoch = 4
LR = 0.1
image_size = 224
number_of_classes = 10
clipping_value = 512

np.random.seed(2)
torch.manual_seed(2)

n_cpu = int(os.cpu_count()*0.5)
n_cpu = 8

class Average_Meter(object):
    """Computes and stores the average and current value"""

    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0.0
        self.avg = 0.0
        self.sum = 0.0
        self.count = 0

    def update(self, val, n):
        if n &gt; 0:
            self.val = val
            self.sum += val * n
            self.count += n
            self.avg = self.sum / self.count


class Sum_Meter(object):
    """Computes and stores the sum and current value"""

    def __init__(self):
        self.reset()

    def reset(self):
        self.val = 0.0
        self.avg = 0.0
        self.sum = 0.0
        self.count = 0

    def update(self, val, n):
        if n &gt; 0:
            self.val = val
            self.sum += val
            self.count += n
            self.avg = self.sum / self.count


def accuracy(output, target, topk=(1,)):
    """Computes the accuracy over the k top predictions for the specified values of k"""
    with torch.no_grad():
        maxk = max(topk)
        batch_size = target.size(0)

        _, pred = output.topk(maxk, 1, True, True)
        pred = pred.t()
        correct = pred.eq(target.view(1, -1).expand_as(pred))

        res = []
        for k in topk:
            correct_k = correct[:k].reshape(-1).float().sum(0, keepdim=True)
            res.append(correct_k.mul_(1.0 / batch_size))
        return res


transform = transforms.Compose([
    transforms.Resize(size=(image_size, image_size),
                      interpolation=PIL.Image.BICUBIC),
    transforms.ToTensor(),
    transforms.Normalize(mean=[0.5, 0.5, 0.5], std=[0.5, 0.5, 0.5])])

trainset = torchvision.datasets.CIFAR10(
    root='./data', train=True,  download=False, transform=transform)
train_loader = torch.utils.data.DataLoader(
    trainset, batch_size=batch_size, shuffle=True, num_workers=n_cpu)

testset = torchvision.datasets.CIFAR10(
    root='./data', train=False, download=False, transform=transform)
val_loader = torch.utils.data.DataLoader(
    testset, batch_size=batch_size, shuffle=False, num_workers=n_cpu)


class MyResNet18(ResNet):
    def __init__(self, num_classes):
        super(MyResNet18, self).__init__(BasicBlock,
                                         [2, 2, 2, 2], num_classes=num_classes)

    def _forward_impl(self, x: Tensor) -&gt; Tensor:
        x = self.conv1(x)
        x = self.bn1(x)
        x = self.relu(x)
        x = self.maxpool(x)
        x = self.layer1(x)
        x = self.layer2(x)
        x = self.layer3(x)
        x = self.layer4(x)
        x = self.avgpool(x)
        FV = torch.flatten(x, 1)
        Logit = self.fc(FV)
        return FV, Logit


model = MyResNet18(num_classes=number_of_classes)

best_val = 0.0

optimizer = torch.optim.Adam(model.parameters(), lr=LR)

device = torch.device('cuda')

flag_multi_gpu = True

if flag_multi_gpu and torch.cuda.device_count() &gt; 1:
    print("Let's use", torch.cuda.device_count(), "GPUs!")
    model = torch.nn.DataParallel(model)
model.to(device)

loss_CE = torch.nn.CrossEntropyLoss(reduction='sum').cuda()

torch.backends.cudnn.benchmark = True

losses_train_total = Sum_Meter()
top1_train = Average_Meter()
top5_train = Average_Meter()
losses_val_total = Sum_Meter()
top1_val = Average_Meter()
top5_val = Average_Meter()

for epoch in range(max_number_of_epoch):
    print("\nepoch = ", epoch + 1)
    losses_train_total.reset()
    top1_train.reset()
    top5_train.reset()
    losses_val_total.reset()
    top1_val.reset()
    top5_val.reset()
    t1 = time.time()
    model.train()
    for i, (x, y) in enumerate(train_loader, 0):
        print("iter %05d: -------DEBUG0------" % i)
        optimizer.zero_grad()
        print("iter %05d: -------DEBUG1------" % i)
        x = x.cuda()
        print("iter %05d: -------DEBUG2------" % i)
        y = y.detach().clone().long().cuda()
        print("iter %05d: -------DEBUG3------" % i)
        FV, Logit = model(x)
        print("iter %05d: -------DEBUG4------" % i)
        prec1, prec5 = accuracy(Logit.data, y, topk=(1, 5))
        loss = loss_CE(Logit, y)
        losses_train_total.update(loss.item(), y.size(0))
        top1_train.update(prec1.item(), y.size(0))
        top5_train.update(prec5.item(), y.size(0))
        assert not torch.isnan(loss)
        loss.backward()
        torch.nn.utils.clip_grad_norm_(model.parameters(), clipping_value)
        optimizer.step()
        # print("iter %05d: -------DEBUG6------" % i)
    model.eval()
    # print("iter %05d: -------DEBUG4------" % i)
    with torch.no_grad():
        for i, (x, y) in enumerate(val_loader, 0):
            x = x.cuda()
            y = y.detach().clone().long().cuda()
            # print("iter %05d: -------DEBUG5------" % i)
            FV, Logit = model(x)
            # print("iter %05d: -------DEBUG6------" % i)
            prec1, prec5 = accuracy(Logit.data, y, topk=(1, 5))
            loss = loss_CE(Logit, y)
            # print("iter %05d: -------DEBUG7------" % i)
            losses_val_total.update(loss.item(), y.size(0))
            top1_val.update(prec1.item(), y.size(0))
            top5_val.update(prec5.item(), y.size(0))
            # print("iter %05d: -------DEBUG8------" % i)
    t2 = time.time()
    print("iter %05d: -------DEBUG9------" % i)
    print('train average_loss_total', losses_train_total.avg)
    print('train top1 accuracy', top1_train.avg)
    print('train top5 accuracy ', top5_train.avg)
    print('validation average_loss_total', losses_val_total.avg)
    print('validation top1 accuracy', top1_val.avg)
    print('validation top5 accuracy ', top5_val.avg)
    print("epoch time = ", t2-t1)
    if top1_val.avg &gt; best_val:
        best_val = top1_val.avg
        print("model saved with vallidation top-1 accuracy  =  ", best_val)
        torch.save(model.state_dict(),
                   f"resnet_18_cifar_10_vall_acc_{best_val}.pth.tar")

print('Finished Training')

</code></pre>
<aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/52142">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/52142" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/52142" target="_blank" rel="noopener nofollow ugc">Deadlock in a single machine multi-gpu using dataparlel when cpu is AMD</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2021-02-11" data-time="15:35:39" data-timezone="UTC">03:35PM - 11 Feb 21 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/ROBOTICSENGINEER" target="_blank" rel="noopener nofollow ugc">
          <img alt="ROBOTICSENGINEER" src="https://avatars.githubusercontent.com/u/7628875?v=4" class="onebox-avatar-inline" width="20" height="20">
          ROBOTICSENGINEER
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: multi-gpu
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: cuda
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: deadlock
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: data parallel
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          module: ddp
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## 🐛 Bug

Training CNN (include torchvision resnet18 and timm efficientnet) wi<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">th a single machine and multi-gpu using dataparallel cause deadlock in machines with AMD cpu, while the same code works well in the machines with Intel cpu.

The code run until forward pass, i.e., `output = model(images)` , inside the for loop in the training. It remains in the `model(images)` forever with gpu utilization go to 0% (memory is occupied, not 0),  three cpu cores go to 100%, and other cpu cores go to 0%. The processes PID and GPU mempry usage remains after stopping with `ctrl+c` and `ctrll+z`. The `kill` , `pkill`  , and `fuser -k /dev/nvidia*` commands cause zombie processes, also known as defunct and z state. The zombie processes have the parent pid of 1, so it cannot be killed. The only solution is to reboot the system. 

The code works well in 3 machines with Intel cpu and has this issue in 4 machines with AMD cpu.
We tested on GTX 1080, Titan V, Titan RTX,  Quadro RTX 8000, and RTX 3090. So, it is independent of gpu model.

**Note**: There is similar issue with Distributed Data Parallel (DDP).


## To Reproduce

Steps to reproduce the behavior:

1. use a machine with AMD cpu and multiple NVIDIA gpu
2. Linux, Python3.8, cuda 11.0, pytorch 1.7.1, torchvision 0.8.2
3. write a code to train a resnet18 model in torchvisaion
4. please test both Data Parallel (DP) and Distributed Data Parallel (DP)

## Expected behavior

1. code go to deadlock at forward pass of in the first epoch and the first iteration of training when using AMD cpu.
2. same code work well when using intel cpu

## Environment

#### Intel cpu environment (system 1)

Intel(R) Xeon(R) CPU E5-2699C v4 @ 2.20GHz

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: 9.1.85
GPU models and configuration:
GPU 0: GeForce RTX 3090
GPU 1: GeForce RTX 3090
GPU 2: GeForce RTX 3090
GPU 3: GeForce RTX 3090

Nvidia driver version: 455.45.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```

#### Intel cpu environment (system 2)

Intel(R) Xeon(R) CPU E5-2699A v4 @ 2.40GHz

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN RTX
GPU 1: TITAN RTX
GPU 2: TITAN RTX
GPU 3: Quadro RTX 8000

Nvidia driver version: 450.80.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```

#### Intel cpu environment (system 3)

Intel(R) Xeon(R) CPU E5-2699A v4 @ 2.40GHz

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla K80
GPU 1: Tesla K80
GPU 2: Tesla K80
GPU 3: Tesla K80
GPU 4: Tesla K80
GPU 5: Tesla K80
GPU 6: Tesla K80
GPU 7: Tesla K80
GPU 8: Tesla K80
GPU 9: Tesla K80
GPU 10: Tesla K80
GPU 11: Tesla K80
GPU 12: Tesla K80
GPU 13: Tesla K80
GPU 14: Tesla K80
GPU 15: Tesla K80

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] numpy==1.14.3
[conda] Could not collect
```


#### AMD cpu environment (system 4)

AMD Eng Sample: 100-000000053-04_32/20_N


```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti
GPU 3: GeForce GTX 1080 Ti
GPU 4: GeForce GTX 1080 Ti

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```


#### AMD cpu environment (system 5)

AMD Eng Sample: 100-000000053-04_32/20_N

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: Could not collect

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN V
GPU 1: TITAN V

Nvidia driver version: 455.45.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```


#### AMD cpu environment (system 6)

AMD Opteron(tm) Processor 6380

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: TITAN V
GPU 1: TITAN V

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```



#### AMD cpu environment (system 7)

AMD Opteron(tm) Processor 6380

```
PyTorch version: 1.7.1+cu110
Is debug build: False
CUDA used to build PyTorch: 11.0
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.5 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2

Python version: 3.8 (64-bit runtime)
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: GeForce GTX 1080 Ti
GPU 1: GeForce GTX 1080 Ti
GPU 2: GeForce GTX 1080 Ti

Nvidia driver version: 450.102.04
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.1.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_adv_train.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8.1.0
/usr/local/cuda-11.0/targets/x86_64-linux/lib/libcudnn_ops_train.so.8.1.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip] Could not collect
[conda] Could not collect
```




cc @ngimel @pietern @mrshenli @pritamdamania87 @zhaojuanmao @satgera @rohan-varma @gqchen @aazzolini @osalpekar @jiayisuse @agolynski @SciPioneer @H-Huang @mrzzd @cbalioglu</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Some information I have learned is that there is no such problem with intel CPUs. The solution given in this Issue is to turn off the IOMMU of the motherboard, but this option is what I need, so I want to know if there are other solutions</p>
<p>I know this should not be a pytorch problem, I’m not sure if I can get some solution from the pytorch community</p>

ptrblck: <p>I’m not sure if I understand the issue completely, but IOMMU is a common problem as you’ve pointed out and disabling it could solve the hang. Did you try it and does it work?</p>

MI_N: <p>I haven’t tried this option yet, but this option is what I need, I need to deploy a virtualized platform on the machine</p>