Building pytorch from source with docker image doesnt include mpi

username: Korneel_Van_Mullem
cooked: <p>Hey everyone</p>
<p>I’m trying to build pytorch from source in a docker image so I can include mpi, as described in the documentation.</p>
<p>However, the final result doesnt seem to have mpi enabled.</p>
<p>I used these commands to build:</p>
<p>git clone <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch<br>
sudo make -f docker.Makefile CMAKE_VARS=“USE_MPI=1 MAX_JOBS=15”</p>
<p>But when I test the end result i get this:</p>
<p>root@88ef37db4d2a:/workspace# python<br>
Python 3.11.11 | packaged by conda-forge | (main, Dec  5 2024, 14:17:24) [GCC 13.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
print(“MPI support:”, torch.distributed.is_mpi_available())<br>
MPI support: False</p>
</blockquote>
</blockquote>
</blockquote>
<p>And printing out the pytorch info</p>
<p>root@88ef37db4d2a:/workspace# python<br>
Python 3.11.11 | packaged by conda-forge | (main, Dec  5 2024, 14:17:24) [GCC 13.3.0] on linux<br>
Type “help”, “copyright”, “credits” or “license” for more information.</p>
<blockquote>
<blockquote>
<blockquote>
<p>import torch<br>
torch.<strong>config</strong>.show()<br>
‘PyTorch built with:\n  - GCC 11.4\n  - C++ Version: 201703\n  - Intel(R) MKL-DNN v3.5.3 (Git Hash 66f0cb9eb66affd2da3bf5f8d897376f04aae6af)\n  - OpenMP 201511 (a.k.a. OpenMP 4.5)\n  - NNPACK is enabled\n  - CPU capability usage: AVX512\n  - Build settings: BUILD_TYPE=Release, COMMIT_SHA=eeb57394f93d720bca498c3fa9d167fc7b9cca46, CUDA_VERSION=12.1, CXX_COMPILER=/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.7.0, USE_CUDA=ON, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_EIGEN_FOR_BLAS=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=OFF, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, \n’</p>
</blockquote>
</blockquote>
</blockquote>
<p>Shows that USE_MPI is off</p>
<p>Why does this happen and how do I solve this? Help would be much appreciated</p>
<p>You can find my image at<br>
<a href="https://hub.docker.com/repository/docker/kvmhowest/pytorch/general" class="onebox" target="_blank" rel="noopener nofollow ugc">https://hub.docker.com/repository/docker/kvmhowest/pytorch/general</a></p>

username: Korneel_Van_Mullem
cooked: <p>Update:</p>
<p>I managed to get things working by manually changing the dockerfile to install openmpi into the conda environment:</p>
<p>Like this:</p>
<h1><a name="p-463058-manually-invoke-bash-on-miniconda-script-per-httpsgithubcomcondacondaissues10431-1" class="anchor" href="#p-463058-manually-invoke-bash-on-miniconda-script-per-httpsgithubcomcondacondaissues10431-1"></a>Manually invoke bash on miniconda script per <a href="https://github.com/conda/conda/issues/10431" class="inline-onebox" rel="noopener nofollow ugc">Miniconda3-latest-Linux-x86_64.sh: 494: [[: not found · Issue #10431 · conda/conda · GitHub</a></h1>
<p>RUN chmod +x ~/miniconda.sh &amp;&amp; <br>
bash ~/miniconda.sh -b -p /opt/conda &amp;&amp; <br>
rm ~/miniconda.sh &amp;&amp; <br>
/opt/conda/bin/conda install -y python=${PYTHON_VERSION} openmpi cmake conda-build pyyaml numpy ipython &amp;&amp; <br>
/opt/conda/bin/python -mpip install -r requirements.txt &amp;&amp; <br>
/opt/conda/bin/conda clean -ya</p>