AZ Notebooks - RuntimeError - on Standard_NC6 and Python 3.8 - Pytorch and Tensorflow kernel

Dimitris_Papatsarouc: <p>Hello PyTorch community,</p>
<p>I have some scripts in a working environment in google colab, and as I am working on my Thesis I tried to use Azure Student promo.<br>
I have set up a Standard_NC6 with Pytorch and Tensorflow kernel and I am getting the following error:</p>
<blockquote>
<hr>
<p>RuntimeError Traceback (most recent call last)<br>
Input In [9], in &lt;cell line: 64&gt;()<br>
76 loss_critic = -(torch.mean(critic_real) - torch.mean(critic_fake))<br>
77 critic.zero_grad()<br>
—&gt; 78 loss_critic.backward(retain_graph=True)<br>
79 opt_critic.step()<br>
81 # clip critic weights between -0.01, 0.01</p>
<p>File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/torch/_tensor.py:396, in Tensor.backward(self, gradient, retain_graph, create_graph, inputs)<br>
387 if has_torch_function_unary(self):<br>
388 return handle_torch_function(<br>
389 Tensor.backward,<br>
390 (self,),<br>
(…)<br>
394 create_graph=create_graph,<br>
395 inputs=inputs)<br>
 → 396 torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)</p>
<p>File /anaconda/envs/azureml_py38_PT_TF/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py:173, in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables, inputs)<br>
168 retain_graph = create_graph<br>
170 # The reason we repeat same the comment below is that<br>
171 # some Python versions print out the first line of a multi-line function<br>
172 # calls in the traceback and some print out the last line<br>
 → 173 Variable.<em>execution_engine.run_backward( # Calls into the C++ engine to run the backward pass<br>
174 tensors, grad_tensors</em>, retain_graph, create_graph, inputs,<br>
175 allow_unreachable=True, accumulate_grad=True)</p>
<p>RuntimeError: Unable to find a valid cuDNN algorithm to run convolution</p>
</blockquote>
<p>I tried different versions of pytorch + cu113 and cu116.</p>
<p>The nvdia-smi output is:</p>
<blockquote>
<p>±----------------------------------------------------------------------------+<br>
| NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 |<br>
|-------------------------------±---------------------±---------------------+<br>
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |<br>
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |<br>
| | | MIG M. |<br>
|===============================+======================+======================|<br>
| 0 Tesla K80 On | 00000001:00:00.0 Off | 0 |<br>
| N/A 41C P0 70W / 149W | 880MiB / 11441MiB | 0% Default |<br>
| | | N/A |<br>
±------------------------------±---------------------±---------------------+</p>
<p>±----------------------------------------------------------------------------+<br>
| Processes: |<br>
| GPU GI CI PID Type Process name GPU Memory |<br>
| ID ID Usage |<br>
|=============================================================================|<br>
| 0 N/A N/A 11425 C …eml_py38_PT_TF/bin/python 877MiB |<br>
±----------------------------------------------------------------------------+</p>
</blockquote>
<p>I guess that the problem is about drivers and versions… as in google colab environment it’s working</p>
<p>Thanks,<br>
DP</p>