Sudden training crippling slowdown since 1.9 on GKE

Ariel_Fabiano: <p>Hi all!<br>
After more than 2 days of fighting I’m back at square one with a weird decrease in training speed and I declare myself officially lost!</p>
<p>On the 15th of June I executed for the last time a train script for a DistilMBERT fine-tuning. At that time, speed was ~6 batches/sec (batch size 16), instead this week with the same code, same dataset and same hyperparams, one batch takes 2 full seconds, making it impossible to train with that speed.</p>
<p>Since it might be relevant to the problem I’ll detail my current setup which is on GCP, running in GKE a node with a Tesla V100 (<a href="https://cloud.google.com/kubernetes-engine/docs/how-to/gpus" rel="noopener nofollow ugc">link</a>), in which we install the nvidia driver (version: 450.51.06) using this <a href="https://github.com/GoogleCloudPlatform/container-engine-accelerators/blob/master/nvidia-driver-installer/cos/daemonset-nvidia-v450.yaml" rel="noopener nofollow ugc">daemonset</a> and afterwards, we create pods from it.<br>
In other words, we have a server with the GPU card and we create VMs on it.</p>
<p>So, after comparing as much as I could, this is what changed in the last month:</p>
<ol>
<li>New Pytorch version: 1.9.0 got released</li>
<li>The CUDA driver on the GKE node server went from 10.2 to 11.0</li>
<li>Not sure if the nvidia drivers changed, but it is highly unlikely</li>
</ol>
<p>Torch was installed without specifying any version, so I’m guessing that 1 month ago it had installed 1.8.0 compiled with CUDA 10.2.</p>
<p>Things I have tried so far:</p>
<ul>
<li>torch==1.9.0+cu111 → same issue</li>
<li>torch==1.8.0+cu111 → same issue</li>
<li>torch==1.8.0 (CUDA 10.2) → same issue</li>
<li>upgrading CUDA driver on the VM to 11.1* → same issue</li>
<li>upgrading CUDA driver on the VM to 11.2* → same issue</li>
<li>disabling cudnn backend and increasing batch size as much as possible (to 48) as suggested in git issue 47908 (<em>The speed of pytorch with cudatoolkit 11.0 is slower than cudatoolkit 10.2</em>) → this lead to a minor speed improvement on the first iterations (around 2 batches/sec), but after a few iterations it went back to the current slow speed</li>
</ul>
<p><strong>*</strong> I’m not entirely sure if this has any impact at all, since Pytorch comes with it’s own CUDA+CUDNN compiled, right? Even further, 1 month ago I was not even installing CUDA drivers on the VM, only consuming the nvidia drivers that came through the node.</p>
<p>Things I can’t do for now:</p>
<ul>
<li>Install CUDA 10.2 on the VM</li>
<li>Downgrade CUDA on the GKE node server</li>
</ul>
<p>My main question here is: could the update on the CUDA drivers on the node where the VM is created cause any issue?</p>
<p>I’m not sure what type of logs or screenshots should I share to provide more info, so please, feel free to request anything!</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

eqy: <p>I’m interested in trying to reproduce this on 10.2/11.x. Do you have a minimal example handy (e.g., just the model with random data) that demonstrates the issue?</p>
<p>The driver itself probably wouldn’t be responsible for the issue. But the switch from CUDA 10.2 torch to 11.x very well might be the cause.</p>

Ariel_Fabiano: <p>Yes, sure! I’ll be able to share something on Monday.</p>
<p>I’m starting to believe my code (which was actually developed 1.5 years ago) might not be optimal for the current version of Pytorch. I might be accumulating something somewhere that with the latest updates causes slowdowns, I’m don’t know.</p>
<p>A <strong>BIG</strong> update though: with torch==1.9.0+cu111 on the server with CUDA 11.0 if I set the following:</p>
<pre><code class="lang-auto">torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = True # in the past this was False
torch.backends.cudnn.enabled = False # does it make sense to disable after setting benchmark True?
</code></pre>
<p>The speed is normal again <img src="https://discuss.pytorch.org/images/emoji/apple/thinking.png?v=9" title=":thinking:" class="emoji" alt=":thinking:"></p>

eqy: <p>I’m not sure the settings make sense, especially if <code>cudnn.enabled = False</code> is set. That being said, I’m not even sure what fraction of operations in a transformer like DistilMBERT would be dispatched to cuDNN, which should really only be doing the heavy lifting for convolution-heavy models.</p>