Data copy between GPUs failed.(Tesla A100, cuda11.1, cudnn8.1.0,pytorch1.8)

111253: <p>Hi, everyone. The configuration I use is Tesla A100, cuda11.0, cudnn8.0.4,pytorch1.7.1. But when I run the following code：</p>
<pre><code class="lang-auto">import torch
a = torch.randn(2, 3, device='cuda:0')
print(a)
b = a.to('cuda:1')
print(b)
</code></pre>
<p>The output are:</p>
<pre><code class="lang-auto">tensor([[-2.0747, -0.5964, -0.3556],
        [-0.5845,  0.3389, -0.3725]], device='cuda:0')

tensor([[0., 0., 0.],
        [0., 0., 0.]], device='cuda:1')
</code></pre>
<p>I am now trying to confirm if this problem is caused at the code level (pytroch,cuda,cudnn) or if there is something wrong with the gpu installation. Thanks!</p>

ptrblck: <p>Could you update to the latest stable release (<code>1.8.0</code>) and rerun the code?<br>
Such error might be software or hardware related and it’s hard to tell it just from these symptoms.</p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thank you for your answer. Now I update to the latest stable release 1.8.0, and I run the following code:</p>
<pre><code class="lang-auto">import torch
import torch.utils
import torch.utils.cpp_extension

print(torch.__version__)
print(torch.cuda.is_available())
print(torch.cuda.device_count())
print(torch.cuda.get_device_name(0))
print(torch.utils.cpp_extension.CUDA_HOME)

a = torch.randn(2, 3, device='cuda:0')
print(a)
b = a.to('cuda:1')
print(b)
</code></pre>
<p>The output are:</p>
<pre><code class="lang-auto">1.8.0+cu111
True
8
A100-PCIE-40GB
/usr/local/cuda-11.1
tensor([[-0.9957, -0.9985,  1.1794],
        [-1.4586, -0.0102, -0.0106]], device='cuda:0')
tensor([[0., 0., 0.],
        [0., 0., 0.]], device='cuda:1')
</code></pre>
<p>Below is my driver configuration：<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/3/236255c09fdc7a7e97c37cd9d6e66f5f432d9272.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/236255c09fdc7a7e97c37cd9d6e66f5f432d9272" title="1"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/236255c09fdc7a7e97c37cd9d6e66f5f432d9272_2_395x500.jpeg" alt="1" data-base62-sha1="531onl0HRI6uKrhCC4l1WCXuZeq" width="395" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/236255c09fdc7a7e97c37cd9d6e66f5f432d9272_2_395x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/2/3/236255c09fdc7a7e97c37cd9d6e66f5f432d9272_2_592x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/2/3/236255c09fdc7a7e97c37cd9d6e66f5f432d9272.jpeg 2x" data-dominant-color="1A1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">739×935 169 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And now I use cuda11.1, cudnn8.1.0.</p>
<p>Do you have any more suggestions to help me troubleshoot the cause (software and hardware) of this problem? Thank you very much.</p>

ptrblck: <p>I tried it on a system with 8x A100s and get a valid output:</p>
<pre><code class="lang-python">1.8.0
True
8
A100-SXM4-40GB
/usr/local/cuda
tensor([[ 0.0440,  1.6352, -0.7515],
        [ 1.6543, -0.5374, -0.8127]], device='cuda:0')
tensor([[ 0.0440,  1.6352, -0.7515],
        [ 1.6543, -0.5374, -0.8127]], device='cuda:1')
</code></pre>
<p>Used driver: 460.32.03.<br>
Could you check <code>dmesg</code> for any Xids and post them here?</p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I don’t know how to check dmesg for any Xids <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:">. When I run dmesg, it outputs a lot of information, and Xid is not included in the information.  Should I post all of them here? And what command should I run? Thanks!</p>

ptrblck: <p>You can use <code>dmesg -T | grep xid</code> and check for the most recent one (if there are any).<br>
Don’t post the complete log here, as it should be quite long <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, after running <code>dmesg -T | grep xid</code>, it outputs nothing <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">. Will the difference between A100-PCIE-40GB and A100-SXM4-40GB lead to this problem?</p>
<p>And I found some new and interesting phenomena. When I run the following code：</p>
<pre><code class="lang-auto">import torch

a = torch.randn(2, 3, device='cuda:0')
print(a)
b = torch.ones(2, 3, device='cuda:1')
print(b)
a1 = a.to('cuda:1')
print(a1)
b1 = b.to('cuda:0')
print(b1)
print(a1 is b)
print(b1 is a)
</code></pre>
<p>The output is:</p>
<pre><code class="lang-auto">tensor([[-1.7723, -0.0860, -1.6667],
        [ 3.1190, -0.2531, -0.7271]], device='cuda:0')
tensor([[1., 1., 1.],
        [1., 1., 1.]], device='cuda:1')
tensor([[1., 1., 1.],
        [1., 1., 1.]], device='cuda:1')
tensor([[-1.7723, -0.0860, -1.6667],
        [ 3.1190, -0.2531, -0.7271]], device='cuda:0')
False
False
</code></pre>
<p>The value of a1 is equal to b and the value of b1 is equal to a. I wonder if this phenomenon can provide new clues.</p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> , I just run the cuda-sample <a href="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/p2pBandwidthLatencyTest" rel="noopener nofollow ugc">p2pBandwidthLatencyTest</a> which demonstrates Peer-To-Peer (P2P) data transfers between pairs of GPUs and computes latency and bandwidth., and it outputs:</p>
<pre><code class="lang-auto">[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, A100-PCIE-40GB, pciBusID: 1, pciDeviceID: 0, pciDomainID:0
Device: 1, A100-PCIE-40GB, pciBusID: 24, pciDeviceID: 0, pciDomainID:0
Device: 2, A100-PCIE-40GB, pciBusID: 41, pciDeviceID: 0, pciDomainID:0
Device: 3, A100-PCIE-40GB, pciBusID: 61, pciDeviceID: 0, pciDomainID:0
Device: 4, A100-PCIE-40GB, pciBusID: 81, pciDeviceID: 0, pciDomainID:0
Device: 5, A100-PCIE-40GB, pciBusID: a1, pciDeviceID: 0, pciDomainID:0
Device: 6, A100-PCIE-40GB, pciBusID: c1, pciDeviceID: 0, pciDomainID:0
Device: 7, A100-PCIE-40GB, pciBusID: e1, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=0 CAN Access Peer Device=2
Device=0 CAN Access Peer Device=3
Device=0 CAN Access Peer Device=4
Device=0 CAN Access Peer Device=5
Device=0 CAN Access Peer Device=6
Device=0 CAN Access Peer Device=7
Device=1 CAN Access Peer Device=0
Device=1 CAN Access Peer Device=2
Device=1 CAN Access Peer Device=3
Device=1 CAN Access Peer Device=4
Device=1 CAN Access Peer Device=5
Device=1 CAN Access Peer Device=6
Device=1 CAN Access Peer Device=7
Device=2 CAN Access Peer Device=0
Device=2 CAN Access Peer Device=1
Device=2 CAN Access Peer Device=3
Device=2 CAN Access Peer Device=4
Device=2 CAN Access Peer Device=5
Device=2 CAN Access Peer Device=6
Device=2 CAN Access Peer Device=7
Device=3 CAN Access Peer Device=0
Device=3 CAN Access Peer Device=1
Device=3 CAN Access Peer Device=2
Device=3 CAN Access Peer Device=4
Device=3 CAN Access Peer Device=5
Device=3 CAN Access Peer Device=6
Device=3 CAN Access Peer Device=7
Device=4 CAN Access Peer Device=0
Device=4 CAN Access Peer Device=1
Device=4 CAN Access Peer Device=2
Device=4 CAN Access Peer Device=3
Device=4 CAN Access Peer Device=5
Device=4 CAN Access Peer Device=6
Device=4 CAN Access Peer Device=7
Device=5 CAN Access Peer Device=0
Device=5 CAN Access Peer Device=1
Device=5 CAN Access Peer Device=2
Device=5 CAN Access Peer Device=3
Device=5 CAN Access Peer Device=4
Device=5 CAN Access Peer Device=6
Device=5 CAN Access Peer Device=7
Device=6 CAN Access Peer Device=0
Device=6 CAN Access Peer Device=1
Device=6 CAN Access Peer Device=2
Device=6 CAN Access Peer Device=3
Device=6 CAN Access Peer Device=4
Device=6 CAN Access Peer Device=5
Device=6 CAN Access Peer Device=7
Device=7 CAN Access Peer Device=0
Device=7 CAN Access Peer Device=1
Device=7 CAN Access Peer Device=2
Device=7 CAN Access Peer Device=3
Device=7 CAN Access Peer Device=4
Device=7 CAN Access Peer Device=5
Device=7 CAN Access Peer Device=6

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1     2     3     4     5     6     7
     0       1     1     1     1     1     1     1     1
     1       1     1     1     1     1     1     1     1
     2       1     1     1     1     1     1     1     1
     3       1     1     1     1     1     1     1     1
     4       1     1     1     1     1     1     1     1
     5       1     1     1     1     1     1     1     1
     6       1     1     1     1     1     1     1     1
     7       1     1     1     1     1     1     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1154.84  14.60  14.67  14.33  15.10  15.62  15.94  17.48
     1  21.10 1290.26  15.47  15.56  15.70  15.75  16.02  17.27
     2  15.24  15.12 1177.47  14.75  15.01  15.67  15.89  17.55
     3  15.13  14.92  15.05 1167.79  16.10  16.14  16.37  17.47
     4  15.11  15.22  15.12  21.26 1292.39  17.51  18.07  15.18
     5  15.15  15.27  15.11  18.93  21.74 1292.39  18.05  16.26
     6  15.02  15.60  19.99  21.53  21.61  21.34 1291.32  15.86
     7  14.74  17.48  21.25  21.47  21.58  21.34  21.59 1295.61
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1156.55   2.11   2.78   2.78   2.78   2.78   2.78   2.78
     1   1.24 1291.32   1.62   1.63   1.63   1.33   1.62   1.62
     2   2.40   2.78 1293.46   2.78   2.78   2.78   2.78   2.78
     3   2.41   2.17   2.78 1293.46   2.78   2.78   2.78   2.78
     4   2.14   2.44   2.78   2.78 1293.46   2.78   2.78   2.78
     5   2.33   2.17   2.78   2.78   2.78 1291.32   2.78   2.78
     6   2.40   2.44   2.78   2.78   2.44   2.78 1293.46   2.78
     7   2.14   2.78   2.78   2.78   2.78   2.78   2.78 1293.46
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1180.58  20.51  19.38  19.44  20.11  20.03  20.32  23.85
     1  23.52 526.72  19.13  19.57  19.82  19.82  20.03  29.41
     2  19.88  18.65 1204.24  18.97  19.56  19.41  19.60  29.57
     3  19.53  20.43  19.69 1291.32  20.39  20.29  20.53  23.41
     4  19.79  20.58  21.06  22.56 1306.98  21.44  21.89  20.29
     5  19.89  20.85  20.41  22.24  30.66 1308.08  21.96  21.02
     6  19.60  21.33  19.79  22.55  30.23  30.73 1308.63  20.59
     7  20.16  29.87  21.30  23.69  22.56  25.27  23.09 1306.98
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1      2      3      4      5      6      7
     0 1184.16   3.25   5.43   4.79   4.28   4.83   4.80   4.81
     1   3.25 487.44   2.66   3.24   3.25   3.25   1.87   3.24
     2   4.70   3.24 1309.17   4.82   4.28   4.32   4.80   4.83
     3   3.87   3.23   5.57 1304.26   4.28   4.83   4.29   3.87
     4   3.87   2.65   4.23   5.57 1308.63   4.28   5.45   4.81
     5   4.73   2.65   4.77   4.28   3.90 1308.63   4.80   4.24
     6   4.20   3.23   4.79   4.24   4.28   4.28 1309.72   4.79
     7   4.28   2.65   4.29   5.43   4.83   4.79   4.29 1304.80
P2P=Disabled Latency Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   4.34  23.57  23.59  23.67  22.00  21.84  22.72  21.99
     1  23.53   2.31  21.76  21.65  15.04  18.91  19.76  18.93
     2  23.57  14.54   2.37  20.29  21.53  15.93  18.08  15.33
     3  23.58  21.56  21.56   2.33  16.20  20.58  21.47  20.71
     4  22.91  15.80  15.57  19.92   2.38  20.55  20.56  18.27
     5  22.50  17.00  19.01  17.91  20.55   2.38  20.58  20.54
     6  22.93  14.91  20.48  12.59  20.56  20.58   2.30  19.28
     7  22.30  21.55  13.68  21.55  21.55  21.55  21.55   2.50

   CPU     0      1      2      3      4      5      6      7
     0   3.79  11.84  12.26  11.93  10.96  10.78  10.57  11.15
     1  12.22   3.54  11.48  11.29  10.21  10.22   9.91  10.45
     2  12.05  11.31   3.59  11.17  10.09  10.12   9.83  10.39
     3  11.93  11.12  11.06   3.53  10.00  10.05   9.78  10.21
     4  11.18  10.38  10.46  10.22   3.24   9.45   9.19   9.56
     5  11.12  10.40  10.50  10.28   9.38   3.15   9.21   9.59
     6  10.95  10.18  10.29  10.10   9.18   9.25   3.08   9.46
     7  11.62  10.76  10.86  10.51   9.54   9.53   9.33   3.77
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1      2      3      4      5      6      7
     0   4.35 49204.95 49204.80 49204.80 49204.78 49204.77 49204.79 49204.79
     1 49205.36   2.31 49204.98 49204.97 49205.17 49204.88 49205.20 49205.15
     2 49204.99 49204.85   2.34 49204.81 49204.84 49204.82 49204.76 49204.83
     3 49205.01 49204.90 49204.84   2.33 49204.84 49204.89 49204.86 49204.79
     4 49204.87 49204.77 49204.73 49204.75   2.39 49204.68 49204.75 49204.73
     5 49204.92 49204.78 49204.81 49204.81 49204.78   2.38 49204.80 49204.76
     6 49205.02 49204.82 49204.87 49204.86 49204.85 49204.83   2.29 49204.85
     7 49205.04 49204.88 49204.90 49204.86 49204.82 49204.83 49204.85   2.47

   CPU     0      1      2      3      4      5      6      7
     0   5.10   2.95   3.02   3.08   3.01   3.03   3.01   2.97
     1   2.98   3.78   2.75   2.85   2.87   2.86   2.86   2.91
     2   2.96   2.77   3.81   2.81   3.05   2.83   2.98   2.72
     3   3.02   2.84   2.77   3.72   2.83   2.86   2.83   2.80
     4   2.82   2.60   2.53   2.62   3.38   2.59   2.63   2.53
     5   2.53   2.56   2.53   2.57   2.43   3.46   2.49   2.38
     6   2.66   2.66   2.55   2.51   2.74   2.67   6.27   4.33
     7   2.70   2.62   2.56   2.58   2.67   2.66   2.57   4.59

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
</code></pre>
<p>I wonder if this results can provide new clues.</p>

ptrblck: <p>Thanks for the input.<br>
I’ll follow up with internal teams on what the next debug steps are.<br>
It seems that the communication is somehow failing, but I’m not sure what might be causing this.</p>
<p>Could you post more information about your system? I.e. is it a custom workstation or a specific pre-configured server?</p>
<p>EDIT:<br>
Are you seeing the same issue using different devices?<br>
E.g. <code>cuda:0</code> to <code>cuda:2</code> etc.?<br>
Also, for the failing case, could you add <code>torch.cuda.synchronize()</code> before printing <code>b</code>?</p>

ptrblck: <p>I also cannot reproduce this issue on a node with 8x A100 PCIE.<br>
Could you check, if ACS is turned off via:</p>
<pre><code class="lang-auto">lspci -vvvv | grep ACSCtl:
</code></pre>
<p>It should return a lot of <code>ACSCtl:	SrcValid-</code>, where <code>SrcValid-</code> shows that ACS is indeed turned off.</p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, Thank you very much for your kind help <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=12" title=":grinning:" class="emoji" alt=":grinning:" loading="lazy" width="20" height="20">.</p>
<p>1.For the following question:</p>
<blockquote>
<p>Could you post more information about your system? I.e. is it a custom workstation or a specific pre-configured server?</p>
</blockquote>
<p>It 's a custom workstation, which has 2 AMD cpu (EPYC 7742) supporting PCIE4 and 1T memory. lnk and sta info are below:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff5f0a5cea872dc88645132ef87bbbc7395dd69a.png" data-download-href="https://discuss.pytorch.org/uploads/default/ff5f0a5cea872dc88645132ef87bbbc7395dd69a" title="1"><img src="https://discuss.pytorch.org/uploads/default/original/3X/f/f/ff5f0a5cea872dc88645132ef87bbbc7395dd69a.png" alt="1" data-base62-sha1="Ar7ns0CsHzYqSTBbbooaQoSZGqK" width="690" height="232" data-dominant-color="181616"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1</span><span class="informations">1031×348 8.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>2.For the following question:</p>
<blockquote>
<p>Are you seeing the same issue using different devices? E.g. <code>cuda:0</code> to <code>cuda:2</code> etc.?</p>
</blockquote>
<p>When I run the following code:</p>
<pre><code class="lang-auto">import torch
a = torch.randn(2, 3, device='cuda:0')
print(a)
b = a.to('cuda:2')
print(b)
</code></pre>
<p>The outputs are:</p>
<pre><code class="lang-auto">tensor([[-0.2846,  0.5795,  1.2842],
        [ 0.3382, -0.4902, -0.8187]], device='cuda:0')
tensor([[0., 0., 0.],
        [0., 0., 0.]], device='cuda:2')
</code></pre>
<p>3.For the following question:</p>
<blockquote>
<p>Also, for the failing case, could you add <code>torch.cuda.synchronize()</code> before printing <code>b</code> ?</p>
</blockquote>
<p>When I run the following code:</p>
<pre><code class="lang-auto">import torch
a = torch.randn(2, 3, device='cuda:0')
print(a)
b = a.to('cuda:2')
torch.cuda.synchronize()
print(b)
</code></pre>
<p>The outputs are:</p>
<pre><code class="lang-auto">tensor([[ 1.3674, -1.1252, -0.1123],
        [ 0.4165,  0.7612,  0.4003]], device='cuda:0')
tensor([[0., 0., 0.],
        [0., 0., 0.]], device='cuda:2')
</code></pre>
<p>4.When  I run <code>lspci -vvvv | grep ACSCtl:</code>, the outputs are:</p>
<pre><code class="lang-auto">		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
pcilib: sysfs_read_vpd: read failed: Input/output error
pcilib: sysfs_read_vpd: read failed: Input/output error
pcilib: sysfs_read_vpd: read failed: Input/output error
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
pcilib: sysfs_read_vpd: read failed: Input/output error
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-

</code></pre>
<p>There are some <code>pcilib: sysfs_read_vpd: read failed: Input/output error</code> message.</p>

ptrblck: <p>Thank you! I’ll forward it to our internal team to discuss it further.</p>

111253: <p>Thank you very much and look forward to your reply <img src="https://discuss.pytorch.org/images/emoji/apple/relaxed.png?v=9" title=":relaxed:" class="emoji" alt=":relaxed:">.</p>

111253: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,  Any progress so far? I look forward to your reply.</p>

ptrblck: <p>No, we would have to wait until work starts again on Monday. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

111253: <p>Sorry, it is Monday now in China <img src="https://discuss.pytorch.org/images/emoji/apple/joy.png?v=9" title=":joy:" class="emoji" alt=":joy:">.  Have a great weekend.</p>

ptrblck: <p>Sorry for the late reply, but I just revisited this topic by chance and took another look at the last outputs.<br>
Could you check the <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">NCCL troubleshooting</a> and disable ACS?</p>