Different training behavior on different machines

Joseph_Assaker: <p>Hello,</p>
<p>I have written some code to implement an architecture idea (composed of Convolutional blocks, Transformer blocks and MLP blocks), and the code trains perfectly well (i.e., smooth loss curve) both on my portable machine (equipped with an NVIDIA GeForce RTX 3080 Laptop GPU) as well as on a VM I have access to (equipped with a GRID V100D-16Q), let’s call this VM1. When I copied this exact same code however to a new VM I got access to, equipped with A100 GPUs - let’s call this one VM2, and ran my training code there, the training wasn’t successful. I made sure that the same code is running with the exact same training data, same hyper-parameters, still training on a single GPU, etc. What was happening more specifically was that the loss started to decrease initially, but after a couple of epochs the loss decrease becomes so slow that it’s almost constant.</p>
<p>Initially, I thought it was a problem of badly setting hyper-parameters, but after trying many configurations I scraped this possibility. Then, my next intuition was that maybe the different Pytorch versions were the culprit here, as I was using PyTorch 1.10 locally and on my current VM1 and only PyTorch 1.11 is supported on the new VM2. So I installed PyTorch 1.11 on VM1, and the training proceeded normally. So it also doesn’t seem to be the version of PyTorch that’s responsible here. I also tried several other test such as setting the following two lines of code in the hope of reducing nondeterminism, but still to no avail:</p>
<p>torch.backends.cudnn.benchmark = False<br>
torch.use_deterministic_algorithms(True, warn_only=True)</p>
<p>And for your reference, here are the differences in the progress of the loss between my local machine (similar to VM1) and the new A100-equipped VM2:</p>
<p>Local Machine (NVIDIA GeForce RTX 3080 Laptop GPU) and current VM1 (GRID V100D-16Q):<br>
12.15, 10.00, 9.18, 8.42, 7.82, 7.38, 7.03, 6.77, 6.56, 6.38<br>
New VM2 (A100 GPU):<br>
14.62, 12.88, 12.51, 12.17, 11.96, 11.76, 11.64, 11.52, 11.47, 11.44<br>
Even after 100 epochs, the loss remains around 8 on this VM (with metrics close to 0 in my case).</p>
<p>So notice that I’m not talking about some simple noise or small difference in performance, I’m talking about going from a perfectly smooth training to no learning at all by simply changing machines.</p>
<p>Would someone have any idea why this might be happening? Could it be the different versions of CUDA? Or could it even be in relation to the difference in CPUs? as the A100 GPUs are set up with AMD CPUs rather than Intel CPUs? Any help or direction would be greatly appreciated.</p>

KFrank: <p>Hi Joseph!</p>
<aside class="quote no-group quote-modified" data-username="Joseph_Assaker" data-post="1" data-topic="162225" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_assaker/48/46872_2.png" class="avatar"> Joseph_Assaker:</div>
<blockquote>
<p>equipped with A100 GPUs - let’s call this one VM2, and ran my training code there, the training wasn’t successful.<br>
…<br>
only PyTorch 1.11 is supported on the new VM2.</p>
</blockquote>
</aside>
<p>This could well be due to the <a href="https://pytorch.org/docs/1.12/notes/cuda.html#tensorfloat-32-tf32-on-ampere-devices" rel="noopener nofollow ugc">TensorFloat-32</a> reduced-precision floating-point<br>
format supported on Ampere GPUs.  Prior to pytorch version 1.12, this was<br>
silently enabled by default.  For version 1.12 it is (silently) enabled by default<br>
for convolutions, but not for explicit <code>matmul()</code>s.</p>
<p>This reduced precision is not supposed to screw up training (and does<br>
speed things up), but, depending on your use case, it possibly could.</p>
<p>Try setting <code>torch.backends.cuda.matmul.allow_tf32</code> and / or<br>
<code>torch.backends.cudnn.allow_tf32</code> to <code>False</code> and see if that fixes<br>
training on your A100s.</p>
<p>Best.</p>
<p>K. Frank</p>

Joseph_Assaker: <p>Hello <a class="mention" href="/u/kfrank">@KFrank</a> !</p>
<p>First off, thank you for taking the time to comment on my issue!</p>
<p>That’s an interesting hypothesis and I tried setting <code>torch.backends.cudnn.allow_tf32</code> to <code>False</code> as suggested, unfortunately however this doesn’t seem to affect the problematic training behavior at all.</p>
<p>Best,</p>
<p>Joseph Assaker.</p>

KFrank: <p>Hi Joseph!</p>
<aside class="quote no-group" data-username="Joseph_Assaker" data-post="3" data-topic="162225" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_assaker/48/46872_2.png" class="avatar"> Joseph_Assaker:</div>
<blockquote>
<p>I tried setting <code>torch.backends.cudnn.allow_tf32</code> to <code>False</code> as suggested,</p>
</blockquote>
</aside>
<p>Did you also try setting <code>torch.backends.cuda.matmul.allow_tf32</code> to <code>False</code>?</p>
<p>Best.</p>
<p>K. Frank</p>

Joseph_Assaker: <p>Hello <a class="mention" href="/u/kfrank">@KFrank</a> ,</p>
<p>I just tried setting <code>torch.backends.cuda.matmul.allow_tf32</code> also to <code>False</code>, but unfortunately it also didn’t affect the training.</p>
<p>Thanks again for your intention to help!</p>
<p>Best,</p>
<p>Joseph Assaker.</p>

KFrank: <p>Hi Joseph!</p>
<aside class="quote no-group" data-username="Joseph_Assaker" data-post="5" data-topic="162225" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/joseph_assaker/48/46872_2.png" class="avatar"> Joseph_Assaker:</div>
<blockquote>
<p>I just tried setting <code>torch.backends.cuda.matmul.allow_tf32</code> also to <code>False</code>, but unfortunately it also didn’t affect the training.</p>
</blockquote>
</aside>
<p>Okay, it sounds like there could be some problem here.</p>
<p>Just to confirm, your VM1 and VM2 are running the same version of pytorch<br>
and get non-trivially different results running the same script, even with both<br>
<code>allow_tf32</code> flags set to <code>False</code>.  Correct?</p>
<p>Narrow things down to the <em>simplest</em> computation that differs on the two systems<br>
by more than floating-point round-off error. This could be a single forward pass<br>
or even just an example <code>matmul()</code>.  (Make sure that you initialize any random<br>
number seeds to the same well-defined values on both systems.)</p>
<p>The best would be if you could create a small, fully-self-contained, runnable<br>
script that reproduces the issue, together with its output from both VM1 and<br>
VM2.</p>
<p>Also, post the results of <code>python -m torch.utils.collect_env</code> for both of<br>
the systems to see if the experts might spot something fishy.</p>
<p>Best.</p>
<p>K. Frank</p>

Joseph_Assaker: <aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="162225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Narrow things down to the <em>simplest</em> computation that differs on the two systems<br>
by more than floating-point round-off error. This could be a single forward pass<br>
or even just an example <code>matmul()</code>. (Make sure that you initialize any random<br>
number seeds to the same well-defined values on both systems.)</p>
</blockquote>
</aside>
<p>That’s exactly what I’ll try to do next.</p>
<aside class="quote no-group" data-username="KFrank" data-post="6" data-topic="162225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p>Also, post the results of <code>python -m torch.utils.collect_env</code> for both of<br>
the systems to see if the experts might spot something fishy.</p>
</blockquote>
</aside>
<p>Here is the output on my portable machine (code works fine):</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.10.2+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
Clang version: Could not collect
CMake version: Could not collect
Libc version: N/A

Python version: 3.6.8 (tags/v3.6.8:3c6b436a57, Dec 24 2018, 00:16:47) [MSC v.1916 64 bit (AMD64)] (64-bit runtime)
Python platform: Windows-10-10.0.19041-SP0
Is CUDA available: True
CUDA runtime version: 11.6.112
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 3080 Laptop GPU
Nvidia driver version: 512.36
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.10.2+cu113
[pip3] torchaudio==0.10.2+cu113
[pip3] torchvision==0.11.3+cu113
[conda] Could not collect
</code></pre>
<p>Here is the output produced on VM1 (code works fine):</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.11.0+cu102
Is debug build: False
CUDA used to build PyTorch: 10.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.4 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: Could not collect
CMake version: version 3.16.3
Libc version: glibc-2.31

Python version: 3.8.10 (default, Mar 15 2022, 12:22:08)  [GCC 9.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-110-generic-x86_64-with-glibc2.29
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration: GPU 0: GRID V100D-16Q
Nvidia driver version: 470.82.01
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.3.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.3.0
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] numpy==1.23.1
[pip3] torch==1.11.0
[pip3] torchvision==0.12.0
[conda] Could not collect
</code></pre>
<p>And finally, here is the output produced on VM2 (where the code doesn’t work as expected):</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.11.0
Is debug build: False
CUDA used to build PyTorch: 11.2
ROCM used to build PyTorch: N/A

OS: Red Hat Enterprise Linux release 8.4 (Ootpa) (x86_64)
GCC version: (GCC) 8.4.1 20200928 (Red Hat 8.4.1-1)
Clang version: Could not collect
CMake version: version 3.23.1
Libc version: glibc-2.28

Python version: 3.9.12 | packaged by conda-forge | (main, Mar 24 2022, 23:25:59)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.18.0-305.57.1.el8_4.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 11.2.67
GPU models and configuration: GPU 0: NVIDIA A100-SXM4-80GB
Nvidia driver version: 510.73.08
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] efficientnet-pytorch==0.6.3
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.2
[pip3] nvidia-dlprof-pytorch-nvtx==1.8.0
[pip3] pytorch-fast-transformers==0.4.0
[pip3] pytorch-ignite==0.4.8
[pip3] pytorch-lightning==1.6.1
[pip3] pytorch-msssim==0.2.1
[pip3] pytorch-pfn-extras==0.5.8
[pip3] pytorch3d==0.7.0
[pip3] pytorchvideo==0.1.5
[pip3] segmentation-models-pytorch==0.2.1
[pip3] torch==1.11.0
[pip3] torch-cluster==1.6.0
[pip3] torch-geometric==2.0.4
[pip3] torch-points-kernels==0.6.10
[pip3] torch-scatter==2.0.9
[pip3] torch-sparse==0.6.13
[pip3] torch-spline-conv==1.2.1
[pip3] torch-tb-profiler==0.4.0
[pip3] torchaudio==0.11.0+820b383
[pip3] torchio==0.18.76
[pip3] torchmetrics==0.8.0
[pip3] torchsparse==1.4.0
[pip3] torchtext==0.12.0a0+d7a34d6
[pip3] torchvision==0.12.0a0
[pip3] torchviz==0.0.2
[conda] efficientnet-pytorch      0.6.3                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.21.2           py39hdbf815f_0    conda-forge
[conda] nvidia-dlprof-pytorch-nvtx 1.8.0                    pypi_0    pypi
[conda] pytorch-fast-transformers 0.4.0                    pypi_0    pypi
[conda] pytorch-ignite            0.4.8                    pypi_0    pypi
[conda] pytorch-lightning         1.6.1                    pypi_0    pypi
[conda] pytorch-msssim            0.2.1                    pypi_0    pypi
[conda] pytorch-pfn-extras        0.5.8                    pypi_0    pypi
[conda] pytorch3d                 0.7.0                    pypi_0    pypi
[conda] pytorchvideo              0.1.5                    pypi_0    pypi
[conda] segmentation-models-pytorch 0.2.1                    pypi_0    pypi
[conda] torch                     1.11.0                   pypi_0    pypi
[conda] torch-cluster             1.6.0                    pypi_0    pypi
[conda] torch-geometric           2.0.4                    pypi_0    pypi
[conda] torch-points-kernels      0.6.10                   pypi_0    pypi
[conda] torch-scatter             2.0.9                    pypi_0    pypi
[conda] torch-sparse              0.6.13                   pypi_0    pypi
[conda] torch-spline-conv         1.2.1                    pypi_0    pypi
[conda] torch-tb-profiler         0.4.0                    pypi_0    pypi
[conda] torchaudio                0.11.0+820b383           pypi_0    pypi
[conda] torchio                   0.18.76                  pypi_0    pypi
[conda] torchmetrics              0.8.0                    pypi_0    pypi
[conda] torchsparse               1.4.0                    pypi_0    pypi
[conda] torchtext                 0.12.0a0+d7a34d6          pypi_0    pypi
[conda] torchvision               0.12.0a0                 pypi_0    pypi
[conda] torchviz                  0.0.2                    pypi_0    pypi
</code></pre>
<p>If anyone can spot something worthy to try out from the produced output, please let me know!</p>
<p>Joseph Assaker.</p>

Javier_Grau: <p>I am experiencing the same on a very similar setup. In my case:</p>
<p>VM1 = RTX 3080, with pytorch=1.9, cuda=11.0 (Windows)<br>
VM2 = A100, with pytorch=1.13, cuda=11.8       (Linux)</p>
<p>Same code, data, hyperparams produce different training curves. VM2 being the worst.</p>
<p>In pytorch I checked weight initalization and optimizer defaults across the two versions and found that they are the same.</p>
<p>Is it the OS? Cuda versions?</p>
<p>This issue might be more critical than it looks like, many people work cross-platform when scaling up their projects.</p>

ptrblck: <aside class="quote no-group" data-username="Javier_Grau" data-post="8" data-topic="162225">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/javier_grau/48/58460_2.png" class="avatar"> Javier_Grau:</div>
<blockquote>
<p>Is it the OS? Cuda versions?</p>
</blockquote>
</aside>
<p>It could be anything since you are using entirely different setups, so try to align at least some of the libraries to narrow down the issue further.</p>