DDP training on RTX 4090 (ADA, cu118)

nicolaspanel: <p>Hi,<br>
DDP training hangs with 100% CPU and no progress when using multiple RTX 4090s. Torch get stuck at</p>
<pre><code class="lang-auto">  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 240, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method='spawn')
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 198, in start_processes
    while not context.join():
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 109, in join
    ready = multiprocessing.connection.wait(
  File "/usr/lib/python3.8/multiprocessing/connection.py", line 931, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.8/selectors.py", line 415, in select
    fd_event_list = self._selector.poll(timeout)
</code></pre>
<p>NOTE: I’m using <a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:22.11-py3</a> container, that comes with torch==1.13.0a0+936e930<br>
NOTE: training on a single GPU works fine</p>
<p>Have anyone found a workaround ?<br>
Best</p>

JamesDickens: <p>What python version are you using?</p>

nicolaspanel: <aside class="quote no-group" data-username="james">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/j/bc8723/48.png" class="avatar"> james:</div>
<blockquote>
<p>What python version are you using?</p>
</blockquote>
</aside>
<p><a href="https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:22.11-py3</a> comes with <code>python 3.8.10</code>. I have the same behavior with <code>python 3.10.9</code></p>

nicolaspanel: <p>After further investigation the problem was due to NCCL backend trying to use peer to peer (P2P) transport.<br>
Forcing <code>NCCL_P2P_DISABLE=1</code> fixed the issue <img src="https://discuss.pytorch.org/images/emoji/apple/+1.png?v=12" title=":+1:" class="emoji" alt=":+1:" loading="lazy" width="20" height="20"></p>

Busby: <p>Thanks, this solves my problem with RTX4090.</p>

Tim_Hanson: <p><code>export NCCL_P2P_DISABLE=1</code> sorta works for models like <a href="https://github.com/The-AI-Summer/pytorch-ddp" class="inline-onebox" rel="noopener nofollow ugc">GitHub - The-AI-Summer/pytorch-ddp: code for the ddp tutorial</a>.  (DDP works, slowly; DP gives NaN loss).<br>
Yet for the life of me I cannot get it working on my models (~= CLIP transformer).  If NCCL is enabled, it hangs with 100% volatile GPU utilization, but the processes can be killed with ^C or kill -9.  If NCCL is disabled, it hard freezes the system.</p>
<p>This was working perfectly well a few days ago on two 2080Ti with otherwise identical hardware.  Model trains fine on either one of the single 4090s.  IOMMU is disabled in BIOS.  memtest good, gpu_burn reports no errors either; hardware seems fine.</p>
<p>These GPUs need nvidia-driver &gt;= 520, (using 525.78.01) which comes with cuda 12.0.  (Related issue: torch_compile also doesn’t work b/c they need sm_89 etc).  I might just train on one GPU until the new hardware bugs get ironed out …</p>

greg_warzecha: <p>I spend more than 80h to debugging where is a problem.</p>
<p>I have a 4x 4090 on WRX-80SE, and second with 7x 4090, both the same.<br>
Cards do not cooperate without NCCL_P2P_DISABLE=1.</p>
<p>But when you enable NCCL_P2P_DISABLE=1, then internal transfer between cards is ~2GB/s, what is EXTREMELY slow. 8x A100 have ~600GB/s p2p communication.</p>
<p>Probably Nvidia blocked p2p connection between 4090, what completely disable that card to use them for training AI models that do not fit in 24GB.</p>
<p>I would be super grateful if someone have a solution, or other configuration on EPYC or Intel that works.</p>

goof: <p>Ran into the same problem on a dual 4090 system.  Tried the latest drivers from nVidia and the 525 available from stock Ubuntu.  Disabling P2P makes it “work”, but then performance is 1it/s compared to 4it/s on my otherwise equivalent dual 3090 system.</p>
<p>The CUDA p2p sample runs without any errors or stalls (<a href="https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/p2pBandwidthLatencyTest" class="inline-onebox" rel="noopener nofollow ugc">cuda-samples/Samples/5_Domain_Specific/p2pBandwidthLatencyTest at master · NVIDIA/cuda-samples · GitHub</a>), but reports atrocious bandwidth between GPUs.</p>
<p>I tried disabling ACS in my BIOS as suggested by (<a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#gpu-direct" class="inline-onebox" rel="noopener nofollow ugc">Troubleshooting — NCCL 2.16.2 documentation</a>), to no available.  My dual 3090 system has ACS enabled and it doesn’t have any issues, so I doubt that’s related anyway.</p>
<p>2x Gigabyte 4090<br>
ASUS Prime X570-Pro (Latest BIOS)<br>
Ryzen 5900X<br>
Ubuntu 20.04.1</p>

ptrblck: <p>Could you post the p2p sample outputs and run additional NCCL tests from <a href="https://github.com/NVIDIA/nccl-tests">this repository</a>, please?<br>
Disabling p2p should not be necessary and I would like to try rebuilding a test system close to your setup to debug the issue.</p>
<p>CC <a class="mention" href="/u/greg_warzecha">@greg_warzecha</a></p>

greg_warzecha: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a> of course! We can schedule a video call if you want.</p>
<p>Now system freeze during NCCL test, without NCCL_P2P_DISABLE=1.<br>
NCCL freezing whole system, after showing a headers, usually system are completly unable to response.</p>
<p>Output with NCCL_P2P_DISABLE=1</p>
<p>ubuntu@g1:~/nvidia/nccl-tests$ ./build/all_reduce_perf -b 8 -e 256M -f 2 -g 2</p>
<h1>
<a name="nthread-1-ngpus-2-minbytes-8-maxbytes-268435456-step-2factor-warmup-iters-5-iters-20-agg-iters-1-validation-1-graph-0-1" class="anchor" href="#nthread-1-ngpus-2-minbytes-8-maxbytes-268435456-step-2factor-warmup-iters-5-iters-20-agg-iters-1-validation-1-graph-0-1"></a>nThread 1 nGpus 2 minBytes 8 maxBytes 268435456 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0</h1>
<h1><a name="h-2" class="anchor" href="#h-2"></a></h1>
<h1>
<a name="using-devices-3" class="anchor" href="#using-devices-3"></a>Using devices</h1>
<h1>
<a name="rank-0-group-0-pid-7064-on-g1-device-0-0x41-nvidia-geforce-rtx-4090-4" class="anchor" href="#rank-0-group-0-pid-7064-on-g1-device-0-0x41-nvidia-geforce-rtx-4090-4"></a>Rank  0 Group  0 Pid   7064 on         g1 device  0 [0x41] NVIDIA GeForce RTX 4090</h1>
<h1>
<a name="rank-1-group-0-pid-7064-on-g1-device-1-0x61-nvidia-geforce-rtx-4090-5" class="anchor" href="#rank-1-group-0-pid-7064-on-g1-device-1-0x61-nvidia-geforce-rtx-4090-5"></a>Rank  1 Group  0 Pid   7064 on         g1 device  1 [0x61] NVIDIA GeForce RTX 4090</h1>
<h1><a name="h-6" class="anchor" href="#h-6"></a></h1>
<h1>
<a name="out-of-place-in-place-7" class="anchor" href="#out-of-place-in-place-7"></a>out-of-place                       in-place</h1>
<h1>
<a name="size-count-type-redop-root-time-algbw-busbw-wrong-time-algbw-busbw-wrong-8" class="anchor" href="#size-count-type-redop-root-time-algbw-busbw-wrong-time-algbw-busbw-wrong-8"></a>size         count      type   redop    root     time   algbw   busbw <span class="hashtag">#wrong</span>     time   algbw   busbw <span class="hashtag">#wrong</span>
</h1>
<h1>
<a name="b-elements-us-gbs-gbs-us-gbs-gbs-9" class="anchor" href="#b-elements-us-gbs-gbs-us-gbs-gbs-9"></a>(B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)</h1>
<pre><code>       8             2     float     sum      -1     6.72    0.00    0.00      0     6.64    0.00    0.00      0
      16             4     float     sum      -1     6.61    0.00    0.00      0     6.78    0.00    0.00      0
      32             8     float     sum      -1     6.59    0.00    0.00      0     6.76    0.00    0.00      0
      64            16     float     sum      -1     6.90    0.01    0.01      0     6.83    0.01    0.01      0
     128            32     float     sum      -1     6.92    0.02    0.02      0     6.68    0.02    0.02      0
     256            64     float     sum      -1     7.03    0.04    0.04      0     6.88    0.04    0.04      0
     512           128     float     sum      -1     7.00    0.07    0.07      0     6.98    0.07    0.07      0
    1024           256     float     sum      -1     7.29    0.14    0.14      0     7.17    0.14    0.14      0
    2048           512     float     sum      -1     7.40    0.28    0.28      0     7.14    0.29    0.29      0
    4096          1024     float     sum      -1     7.69    0.53    0.53      0     7.63    0.54    0.54      0
    8192          2048     float     sum      -1     8.54    0.96    0.96      0     8.42    0.97    0.97      0
   16384          4096     float     sum      -1    10.31    1.59    1.59      0    10.30    1.59    1.59      0
   32768          8192     float     sum      -1    14.18    2.31    2.31      0    14.09    2.33    2.33      0
   65536         16384     float     sum      -1    21.40    3.06    3.06      0    21.35    3.07    3.07      0
  131072         32768     float     sum      -1    31.69    4.14    4.14      0    31.62    4.15    4.15      0
  262144         65536     float     sum      -1    48.13    5.45    5.45      0    47.92    5.47    5.47      0
  524288        131072     float     sum      -1    77.00    6.81    6.81      0    75.53    6.94    6.94      0
 1048576        262144     float     sum      -1    137.5    7.63    7.63      0    136.8    7.67    7.67      0
 2097152        524288     float     sum      -1    262.2    8.00    8.00      0    262.4    7.99    7.99      0
 4194304       1048576     float     sum      -1    513.7    8.16    8.16      0    514.8    8.15    8.15      0
 8388608       2097152     float     sum      -1   1034.2    8.11    8.11      0   1034.2    8.11    8.11      0
16777216       4194304     float     sum      -1   2078.5    8.07    8.07      0   2080.2    8.07    8.07      0
33554432       8388608     float     sum      -1   4175.0    8.04    8.04      0   4164.4    8.06    8.06      0
67108864      16777216     float     sum      -1   8332.2    8.05    8.05      0   8330.4    8.06    8.06      0
</code></pre>
<p>134217728      33554432     float     sum      -1    16678    8.05    8.05      0    16677    8.05    8.05      0<br>
268435456      67108864     float     sum      -1    33378    8.04    8.04      0    33380    8.04    8.04      0</p>
<h1>
<a name="out-of-bounds-values-0-ok-10" class="anchor" href="#out-of-bounds-values-0-ok-10"></a>Out of bounds values : 0 OK</h1>
<h1>
<a name="avg-bus-bandwidth-375734-11" class="anchor" href="#avg-bus-bandwidth-375734-11"></a>Avg bus bandwidth    : 3.75734</h1>
<p>Example output WITHOUT: NCCL_P2P_DISABLE=1</p>
<p>ubuntu@g1:~/nvidia/nccl-tests$ ./build/all_reduce_perf -b 8 -e 256M -f 2 -g 2</p>
<h1>
<a name="nthread-1-ngpus-2-minbytes-8-maxbytes-268435456-step-2factor-warmup-iters-5-iters-20-agg-iters-1-validation-1-graph-0-12" class="anchor" href="#nthread-1-ngpus-2-minbytes-8-maxbytes-268435456-step-2factor-warmup-iters-5-iters-20-agg-iters-1-validation-1-graph-0-12"></a>nThread 1 nGpus 2 minBytes 8 maxBytes 268435456 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0</h1>
<h1><a name="h-13" class="anchor" href="#h-13"></a></h1>
<h1>
<a name="using-devices-14" class="anchor" href="#using-devices-14"></a>Using devices</h1>
<h1>
<a name="rank-0-group-0-pid-7064-on-g1-device-0-0x41-nvidia-geforce-rtx-4090-15" class="anchor" href="#rank-0-group-0-pid-7064-on-g1-device-0-0x41-nvidia-geforce-rtx-4090-15"></a>Rank  0 Group  0 Pid   7064 on         g1 device  0 [0x41] NVIDIA GeForce RTX 4090</h1>
<h1>
<a name="rank-1-group-0-pid-7064-on-g1-device-1-0x61-nvidia-geforce-rtx-4090-16" class="anchor" href="#rank-1-group-0-pid-7064-on-g1-device-1-0x61-nvidia-geforce-rtx-4090-16"></a>Rank  1 Group  0 Pid   7064 on         g1 device  1 [0x61] NVIDIA GeForce RTX 4090</h1>
<p>FREEZE</p>
<p>I checking almost everything:</p>
<ul>
<li>WRX80SE motherboard settings,</li>
<li>PCI gen4 or 3,</li>
<li>ACS, IOMMU etc…</li>
</ul>
<p>If you want we will love to schedule a call and stream everything.<br>
Our company calculating a human IgG antibodies using AI, and it’s too expensive for us A100 or H100.</p>

goof: <p>Not sure if you meant me or an earlier poster, but since I’m here:</p>
<p>Running inside <code>sudo docker run --rm --gpus all -it pytorch/pytorch:1.13.1-cuda11.6-cudnn8-devel</code></p>
<p><strong>P2P sample:</strong></p>
<pre><code class="lang-auto">root@5552c3798cde:/cuda-samples/bin/x86_64/linux/release# ./p2pBandwidthLatencyTest 
[P2P (Peer-to-Peer) GPU Bandwidth Latency Test]
Device: 0, NVIDIA GeForce RTX 4090, pciBusID: 4, pciDeviceID: 0, pciDomainID:0
Device: 1, NVIDIA GeForce RTX 4090, pciBusID: 9, pciDeviceID: 0, pciDomainID:0
Device=0 CAN Access Peer Device=1
Device=1 CAN Access Peer Device=0

***NOTE: In case a device doesn't have P2P access to other one, it falls back to normal memcopy procedure.
So you can see lesser Bandwidth (GB/s) and unstable Latency (us) in those cases.

P2P Connectivity Matrix
     D\D     0     1
     0	     1     1
     1	     1     1
Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 910.55   6.63 
     1   6.13 922.37 
Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)
   D\D     0      1 
     0 913.21   6.78 
     1  17.79 922.92 
Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 916.96   8.83 
     1   7.87 924.01 
Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)
   D\D     0      1 
     0 917.77  13.55 
     1  13.55 924.26 
P2P=Disabled Latency Matrix (us)
   GPU     0      1 
     0   1.38  18.07 
     1  18.73   1.41 

   CPU     0      1 
     0   1.84   5.88 
     1   5.69   1.75 
P2P=Enabled Latency (P2P Writes) Matrix (us)
   GPU     0      1 
     0   1.39   1.60 
     1   1.02   1.40 

   CPU     0      1 
     0   1.82   1.46 
     1   1.49   1.78 

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.
</code></pre>
<p><strong>NCCL Tests</strong><br>
Wasn’t sure which of the nccl tests to run or with what options, so I tried the example.</p>
<pre><code class="lang-auto">root@89884fa4021a:/nccl-tests# ./build/all_reduce_perf 
# nThread 1 nGpus 1 minBytes 33554432 maxBytes 33554432 step: 1048576(bytes) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid    946 on 89884fa4021a device  0 [0x04] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
    33554432       8388608     float     sum      -1    18.55  1809.10    0.00      0     0.25  136123.46    0.00      0
# Out of bounds values : 0 OK
# Avg bus bandwidth    : 0 
#

root@89884fa4021a:/nccl-tests# 
root@89884fa4021a:/nccl-tests# 
root@89884fa4021a:/nccl-tests# 
root@89884fa4021a:/nccl-tests# ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 2
# nThread 1 nGpus 2 minBytes 8 maxBytes 134217728 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0
#
# Using devices
#  Rank  0 Group  0 Pid    955 on 89884fa4021a device  0 [0x04] NVIDIA GeForce RTX 4090
#  Rank  1 Group  0 Pid    955 on 89884fa4021a device  1 [0x09] NVIDIA GeForce RTX 4090
#
#                                                              out-of-place                       in-place          
#       size         count      type   redop    root     time   algbw   busbw #wrong     time   algbw   busbw #wrong
#        (B)    (elements)                               (us)  (GB/s)  (GB/s)            (us)  (GB/s)  (GB/s)       
</code></pre>
<p>The first command ran on only one GPU.  The second, as specified by the README, ran on both GPUs and … froze.  I waited several minutes and didn’t see any progress, much like my DDP workload.  Responds to Ctrl+C and immediately quits though.</p>
<p><strong>nvidia-smi</strong></p>
<pre><code class="lang-auto">root@89884fa4021a:/nccl-tests# nvidia-smi
Tue Jan 24 20:38:42 2023       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.78.01    Driver Version: 525.78.01    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:04:00.0 Off |                  Off |
|  0%   44C    P8    26W / 450W |      6MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:09:00.0 Off |                  Off |
|  0%   47C    P8    29W / 450W |     73MiB / 24564MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
</code></pre>

akshu2023: <p>Seems to be AMD specific issue with multiple 4090s NCCL P2P functionality. Unsure who will resolve it when.</p>
<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/parallel-training-with-4-cards-4090-cannot-be-performed-on-amd-5975wx-stuck-at-the-beginning/237813/6">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/parallel-training-with-4-cards-4090-cannot-be-performed-on-amd-5975wx-stuck-at-the-beginning/237813/6" target="_blank" rel="noopener nofollow ugc" title="02:22AM - 24 December 2022">NVIDIA Developer Forums – 24 Dec 22</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/459;"><img src="https://global.discourse-cdn.com/nvidia/original/3X/b/a/ba30443ceea6e42a4c8312c74749fd270a72b048.jpeg" class="thumbnail" width="690" height="459"></div>

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/parallel-training-with-4-cards-4090-cannot-be-performed-on-amd-5975wx-stuck-at-the-beginning/237813/6" target="_blank" rel="noopener nofollow ugc">Parallel training with 4 cards 4090 cannot be performed on AMD 5975WX， stuck...</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Accelerated Computing</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">CUDA</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">CUDA Programming and Performance</span>
        </span>
      </span>
  </div>
</div>

  <p>The motherboard I’m using is Supermicro’s M12SWA-TF, I’m sure the BIOS has been updated to the latest from the official website, and the ASC has been turned off. But my situation is still the same, and I am not sure whether the 4090 supports P2P,...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


akshu2023: <p>Here is another thread - awaiting some progress.</p>
<aside class="onebox discoursetopic" data-onebox-src="https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/2">
  <header class="source">
      <img src="https://global.discourse-cdn.com/nvidia/optimized/2X/0/0372ccc95874f71d7fbff64bbbff6f8c69dd850b_2_32x32.png" class="site-icon" width="32" height="32">

      <a href="https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/2" target="_blank" rel="noopener nofollow ugc" title="08:51PM - 24 January 2023">NVIDIA Developer Forums – 24 Jan 23</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:150/80;"><img src="https://global.discourse-cdn.com/nvidia/original/3X/e/4/e41d27491ee72562cf68c340597ca95f587879bd.png" class="thumbnail" width="150" height="80"></div>

<div class="title-wrapper">
  <h3><a href="https://forums.developer.nvidia.com/t/standard-nvidia-cuda-tests-fail-with-dual-rtx-4090-linux-box/233202/2" target="_blank" rel="noopener nofollow ugc">Standard nVidia CUDA tests fail with dual RTX 4090 Linux box</a></h3>
  <div class="topic-category">
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Graphics / Linux</span>
        </span>
      </span>
      <span class="badge-wrapper bullet">
        <span class="badge-category-bg" style="background-color: #76B900;"></span>
        <span class="badge-category clear-badge">
          <span class="category-name">Linux</span>
        </span>
      </span>
  </div>
</div>

  <p>Hi Vasilii,  Apologies for the delay. Can you please capture a Nvidia bug report from your system.  Please run nvidia-bug-report.sh as root or sudo user and attach the generated nvidia-bug-report.log.gz.  Can you also please provide the make/model...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


iafoss: <p>I’m not sure if it is fully related to the issue, but on my 2x4090 I get the following:</p>
<pre><code class="lang-auto">import torch
v = torch.randn(5, device='cuda:0')
print(v)
print(v.to('cuda:1'))
print(v.to('cpu').to('cuda:1'))
</code></pre>
<p>tensor([ 1.5336,  0.8161, -0.9325, -0.9513,  0.1360], device=‘cuda:0’)<br>
tensor([0., 0., 0., 0., 0.], device=‘cuda:1’)<br>
tensor([ 1.5336,  0.8161, -0.9325, -0.9513,  0.1360], device=‘cuda:1’)</p>
<p><strong>It looks there is a bug (very likely at NVIDIA site) in GPU-to-GPU memory copy. So it sets everything to zeros.</strong> I have the latest NVIDIA driver and tried the latest stable PyTorch as well as Pytorch 2.0 preview.</p>
<p>And here is NVIDIA P2P test output:</p>
<pre><code class="lang-auto">[./simpleP2P] - Starting...
Checking for multiple GPUs...
CUDA-capable device count: 2

Checking GPU(s) for support of peer to peer memory access...
&gt; Peer access from NVIDIA GeForce RTX 4090 (GPU0) -&gt; NVIDIA GeForce RTX 4090 (GPU1) : Yes
&gt; Peer access from NVIDIA GeForce RTX 4090 (GPU1) -&gt; NVIDIA GeForce RTX 4090 (GPU0) : Yes
Enabling peer access between GPU0 and GPU1...
Allocating buffers (64MB on GPU0, GPU1 and CPU Host)...
Creating event handles...
cudaMemcpyPeer / cudaMemcpy between GPU0 and GPU1: 12.61GB/s
Preparing host buffer and memcpy to GPU0...
Run kernel on GPU1, taking source data from GPU0 and writing to GPU1...
Run kernel on GPU0, taking source data from GPU1 and writing to GPU0...
Copy data back to host from GPU0 and verify results...
Verification error @ element 1: val = 0.000000, ref = 4.000000
Verification error @ element 2: val = 0.000000, ref = 8.000000
Verification error @ element 3: val = 0.000000, ref = 12.000000
Verification error @ element 4: val = 0.000000, ref = 16.000000
Verification error @ element 5: val = 0.000000, ref = 20.000000
Verification error @ element 6: val = 0.000000, ref = 24.000000
Verification error @ element 7: val = 0.000000, ref = 28.000000
Verification error @ element 8: val = 0.000000, ref = 32.000000
Verification error @ element 9: val = 0.000000, ref = 36.000000
Verification error @ element 10: val = 0.000000, ref = 40.000000
Verification error @ element 11: val = 0.000000, ref = 44.000000
Verification error @ element 12: val = 0.000000, ref = 48.000000
Disabling peer access...
Shutting down...
Test failed!
</code></pre>

ptrblck: <p>Could you disable p2p via <code>export NCCL_P2P_DISABLE=1</code> and check if this would solve the issue, as you might be running into a driver bug.</p>

iafoss: <p>Thank you for checking. It doesn’t change the behavior. I also disabled IOMMU and SVM in bios.</p>
<pre><code class="lang-auto">import os
os.environ["NCCL_P2P_DISABLE"] = "1"
​import torch

print('Test 1')
v = torch.randn(5, device='cuda:0')
print(v)
print(v.to('cuda:1'))
print(v.to('cpu').to('cuda:1'))

print('Test 2')
v = torch.randn(5, device='cuda:0')
print(v)
print(v.to('cuda:1'))
print(v.to('cpu').to('cuda:1'))
</code></pre>
<p>Test 1<br>
tensor([-0.1360, -1.5022, -1.9172,  0.8753,  0.5528], device=‘cuda:0’)<br>
tensor([0., 0., 0., 0., 0.], device=‘cuda:1’)<br>
tensor([-0.1360, -1.5022, -1.9172,  0.8753,  0.5528], device=‘cuda:1’)<br>
Test 2<br>
tensor([-0.5404, -1.6951, -0.4220, -0.9484,  0.1218], device=‘cuda:0’)<br>
tensor([-0.1360, -1.5022, -1.9172,  0.8753,  0.5528], device=‘cuda:1’)<br>
tensor([-0.5404, -1.6951, -0.4220, -0.9484,  0.1218], device=‘cuda:1’)</p>
<p>It is very likely NVIDIA driver related issue( I just finished building 2x4090 system, and in the initial testing I realized that PyTorch is not working properly with multiple GPUs. Hopefully it will be fixed by NVIDIA soon.</p>

ptrblck: <p>Set the env variable in your terminal, as setting it in the Python script is tricky and fails if it’s set too late.<br>
Based on <a href="https://github.com/NVIDIA/nccl-tests/issues/117">this discussion</a> disabling p2p should help and yes, it seems to be a driver issue.</p>

iafoss: <p>Unfortunately it doesn’t work for me even if I set it in the terminal(</p>
<p>Probably some additional issues may be caused by 670e chipset beyond ones 4090 cards have. As I understand, NCCL_P2P_DISABLE should make the driver behavior to be identical to “.to(‘cpu’).to(‘cuda:1’)” in case of device to device copy because it goes through the shared memory, but somehow my test still gives zeros if I call “.to(‘cuda:1’)”.</p>

ptrblck: <p>Could you check if IO Virtualization is on and disable it as described <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html#pci-access-control-services-acs">here</a>?</p>

iafoss: <p>Thanks. After running “sudo lspci -vvv | grep ACSCtl”, I get the following:</p>
<pre><code class="lang-auto">ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
pcilib: sysfs_read_vpd: read failed: No such device
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
</code></pre>
<p>It might indicate that ACS is disabled, or may I be missing something?</p>