MKL error pytorch on Azure ubuntu VM

username: Chris_Palmer
cooked: <p>I am having a problem after installing the latest pytorch into an Azure Data Science Virtual Machine, via the fast.ai conda environment.</p>
<p>I have CUDA 9 installed, and after following a thread linked to Issue <span class="hashtag">#5099</span> I tried upgrading MKL, but all it offered me was and upgrade from 2018.0.2-1 to 2018.0.2-intel_1 intel, so it doesn’t seem out of date.</p>
<p>I believe I have have the latest version of pytorch, as I’ve just installed it using the conda install process and -c soumith. Unfortunately I cannot verify my torch version as I have an error loading torch, that there is an undefined symbol: mkl_lapack_ao_ssyrdb in /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so. (see below for trace).</p>
<p>Following the Issue <span class="hashtag">#5099</span> thread I also executed echo $LD_LIBRARY_PATH, and got this:<br>
/opt/intel/tbb/lib/intel64/gcc4.7:/opt/intel/lib/intel64:/opt/intel/mkl/lib/intel64:/dsvm/tools/cntk/cntk/lib:/dsvm/tools/cntk/cntk/dependencies/lib::/usr/local/cuda/lib64</p>
<p>Any idea about how to fix this?</p>
<p>`<br>
import torch</p>
<p>ImportError Traceback (most recent call last)<br>
in ()<br>
----&gt; 1 import torch</p>
<p>~/.conda/envs/fastai/lib/python3.6/site-packages/torch/init.py in ()<br>
54 except ImportError:<br>
55 pass<br>
—&gt; 56 from torch._C import *<br>
57<br>
58 all += [name for name in dir(_C)</p>
<p>ImportError: /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so: undefined symbol: mkl_lapack_ao_ssyrdb<br>
`</p>

username: SimonW
cooked: <p>Is the symbol actually in the .so? Can you try <code>nm -an libmkl_gf_lp64 .so | grep mkl_lapack_ao_ssyrdb</code></p>

username: Chris_Palmer
cooked: <p>Thanks Simon, for getting involved!</p>
<p>I tried exactly what you said, but the libmkl_gf_lp64 was not found.</p>
<p>So I tried specifying the path to the one that Pytorch errored on, and the response was Unknown:</p>
<pre><code class="lang-auto">nm -an  /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so | grep mkl_lapack_ao_ssyrdb
U mkl_lapack_ao_ssyrdb
</code></pre>
<p>I tried shortening the value being looked for so I could see what is listed:</p>
<pre><code class="lang-auto">Chris_Palmer@FASTAI:~$ nm -an  /opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so | grep mkl_lapack_ao_
                 U mkl_lapack_ao_cgeqrf
                 U mkl_lapack_ao_cgetrf
                 U mkl_lapack_ao_cgetrfnpi
                 U mkl_lapack_ao_cpotrf
                 U mkl_lapack_ao_cspffrt2
                 U mkl_lapack_ao_cspffrtx
                 U mkl_lapack_ao_dgeqrf
                 U mkl_lapack_ao_dgetrf
                 U mkl_lapack_ao_dgetrfnpi
                 U mkl_lapack_ao_dpotrf
                 U mkl_lapack_ao_dspffrt2
                 U mkl_lapack_ao_dspffrtx
                 U mkl_lapack_ao_dsyrdb
                 U mkl_lapack_ao_inquiry
                 U mkl_lapack_ao_sgeqrf
                 U mkl_lapack_ao_sgetrf
                 U mkl_lapack_ao_sgetrfnpi
                 U mkl_lapack_ao_spotrf
                 U mkl_lapack_ao_sspffrt2
                 U mkl_lapack_ao_sspffrtx
                 U mkl_lapack_ao_ssyrdb
                 U mkl_lapack_ao_zgeqrf
                 U mkl_lapack_ao_zgetrf
                 U mkl_lapack_ao_zgetrfnpi
                 U mkl_lapack_ao_zpotrf
                 U mkl_lapack_ao_zspffrt2
                 U mkl_lapack_ao_zspffrtx
</code></pre>
<p>I looked for all instances of libmkl_gf_lp64.so, and found many candidates.</p>
<p>When I execute nn -an against any of these I get no response coming back… is that bad?</p>
<p>On the most likely candidate I tried again the first part of the name to get near:</p>
<pre><code class="lang-auto">nm -an  /home/Chris_Palmer/.conda/envs/fastai/lib/libmkl_gf_lp64.so | grep mkl_lapack_ao_
U mkl_lapack_ao_inquiry
</code></pre>
<p>If any of this seem to be OK, what’s the optimal way to tell the system to use the correct one?</p>
<pre><code class="lang-auto">/home/Chris_Palmer/.conda/envs/fastai/lib/libmkl_gf_lp64.so
/data/home/Chris_Palmer/.conda/envs/fastai/lib/libmkl_gf_lp64.so
/anaconda/lib/libmkl_gf_lp64.so
/anaconda/pkgs/mkl-2018.0.0-hb491cac_4/lib/libmkl_gf_lp64.so
/anaconda/pkgs/mkl-2018.0.2-intel_1/lib/libmkl_gf_lp64.so
/anaconda/pkgs/mkl-2018.0.2-1/lib/libmkl_gf_lp64.so
</code></pre>

username: SimonW
cooked: <p>Hmm this seems another instance of <a href="https://github.com/pytorch/pytorch/issues/6131">https://github.com/pytorch/pytorch/issues/6131</a>. Can you remove either your system mkl or conda mkl?</p>

username: Chris_Palmer
cooked: <p>Following that link and other it linked to, there seems to be a bit of discussion about these sorts of issues. Most of which related to building Pytorch in situ, MKL, and NNPACK. Do you think its possible that a new version of Pytorch might address these issues?</p>
<p>Regarding your advice, perhaps I’ll try removing the Conda MKL - the system seems to be locked up (I wasn’t permitted to upgrade the system MKL).</p>
<p>Do you know how to determine the MKL version in each directory? (I’ve searched but have found no clear instruction - I only found my CONDA versions by doing the INSTALL again, when it reported the versions to me).</p>
<p>Can you please confirm what exactly I should do? Should I just CONDA UNINSTALL my conda installations?</p>
<p>Is there no PATH setting I can use to force this to see the right information?</p>
<p>Thanks</p>

username: SimonW
cooked: <p>I was not saying that it will fix your problem. I was saying that it worth a try to remove one install. So maybe <code>conda uninstall</code> works, maybe it doesn’t.</p>
<p>I’m not sure if there is any env var setting for this.</p>
<p>Btw, the latest pytorch version is at <code>-c pytorch</code>, and you can check install version by directly looking at the <code>torch/version,py</code></p>

username: Chris_Palmer
cooked: <p>Sure, I will try it.</p>
<p>Actually I am unsure what it meant in my error that the reference was unknown. Is it because something is missing from my system, or that there is a conflict trying to determine where to go for it? Does “undefined symbol: mkl_lapack_ao_ssyrdb” mean that mkl_lapack_ao_ssyrdb should be able to be found somewhere but cannot be? Where would it normally be found? Is it referenced because Pytorch requires it, or just because its mentioned in libmkl_gf_lp64.so?</p>

username: dae
cooked: <p>It looks there’s a backward compatibility issue with your version of mkl <code>mkl-2018.0.2</code> in your output.<br>
You can try to downgrade it to previous version: <code>conda install mkl=2018.0.1</code><br>
And restart the notebook.</p>

username: Chris_Palmer
cooked: <aside class="quote no-group" data-username="dae" data-post="8" data-topic="15689">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/dae/48/4152_2.png" class="avatar"> dae:</div>
<blockquote>
<p>It looks there’s a backward compatibility issue with your version of mkl mkl-2018.0.2 in your output.</p>
</blockquote>
</aside>
<p>Actually, I already had 2018.0.1 when the error occurred - the 2018.0.2 came in when I reinstalled MKL (using conda) as an attempt to resolve the problem.</p>
<p>But I believe that the problem I have is that it is going to the system MKL<br>
/opt/intel/mkl/lib/intel64/libmkl_gf_lp64.so</p>
<p>rather than the one that’s compatible with my anaconda installation<br>
/home/Chris_Palmer/.conda/envs/fastai/lib/libmkl_gf_lp64.so</p>
<p>which I “upgraded” to 2018.0.2 <em>after</em> encountering the error, but which I can revert to 2018.0.1</p>
<p>How can I determine the version of the library in /opt/intel/mkl/lib/intel64 ? I think it’s not possible for me to alter it but it would be nice to know what I am dealing with!</p>
<p>How to force my system to use the correct one?</p>

username: dae
cooked: <p>You can run <code>conda list | grep mkl</code> to see the version you have. If you still can’t figure it out, you might want to try setting up again from scratch with proven working setup.</p>
<p>If you are setting up the environment for fast.ai with latest pytorch on Azure, you can try setting up with the following proven steps</p>
<ol>
<li>Create a new <code>Deep Learning Virtual Machine</code> resource, and choose Linux, new Resource Group (easier to manage later), HDD, NC6</li>
<li>Connect to the VM: <code>ssh username@ip</code>
</li>
<li>Download the setup script: <code>curl http://files.fast.ai/setup/paperspace -o setup.sh</code>
</li>
<li>Remove line <code>sudo rm '/etc/apt/apt.conf.d/*.*'</code> using <code>vim setup.sh</code>
</li>
<li>Make script executable: <code>chmod +x setup.sh</code>
</li>
<li>Execute the script: <code>./setup.sh</code>
</li>
<li>Follow the script and <strong>reboot</strong> after successful installation, everything should be setup with latest installs.</li>
<li>
<strong>Important:</strong> Downgrade mkl back to prior version: <code>conda install mkl=2018.0.1</code>
</li>
</ol>