Distributed pytorch with mpi

ph0123: <p>Hi all ,</p>
<p>I try to run pytorch with distributed system.<br>
I run test1.py as below</p>
<blockquote>
<p>import torch<br>
import torch.distributed as dist<br>
def main(rank, world):<br>
if rank == 0:<br>
x = torch.tensor([1., -1.]) # Tensor of interest<br>
dist.send(x, dst=1)<br>
print(‘Rank-0 has sent the following tensor to Rank-1’)<br>
print(x)<br>
else:<br>
z = torch.tensor([0., 0.]) # A holder for recieving the tensor<br>
dist.recv(z, src=0)<br>
print(‘Rank-1 has recieved the following tensor from Rank-0’)<br>
print(z)<br>
if <strong>name</strong> == ‘<strong>main</strong>’:<br>
dist.init_process_group(backend=‘mpi’)<br>
main(dist.get_rank(), dist.get_world_size())</p>
</blockquote>
<p>Then I run with single machine.</p>
<blockquote>
<p>mpiexec -n 2 python test1.py</p>
</blockquote>
<p>Finally, the error is</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File "test1.py", line 17, in &lt;module&gt;
    dist.init_process_group(backend='mpi')
  File "/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 392, in init_process_group
    timeout=timeout)
  File "/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 452, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have MPI built in")
RuntimeError: Distributed package doesn't have MPI built in
Traceback (most recent call last):
  File "test1.py", line 17, in &lt;module&gt;
    dist.init_process_group(backend='mpi')
  File "/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 392, in init_process_group
    timeout=timeout)
  File "/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 452, in _new_process_group_helper
    raise RuntimeError("Distributed package doesn't have MPI built in")
RuntimeError: Distributed package doesn't have MPI built in
-------------------------------------------------------
Primary job  terminated normally, but 1 process returned
a non-zero exit code. Per user-direction, the job has been aborted.
-------------------------------------------------------
--------------------------------------------------------------------------
mpirun detected that one or more processes exited with non-zero status, thus causing
the job to be terminated. The first process to do so was:

  Process name: [[31741,1],1]
  Exit code:    1
--------------------------------------------------------------------------
</code></pre>
<p>I also installed pytorch with</p>
<blockquote>
<p>pip install torch torchvision</p>
</blockquote>
<p>Please help me.<br>
Thanks,</p>

pritamdamania87: <p>You need to build pytorch from source to enable MPI: <a href="https://pytorch.org/docs/stable/distributed.html#backends-that-come-with-pytorch" rel="nofollow noopener">https://pytorch.org/docs/stable/distributed.html#backends-that-come-with-pytorch</a></p>

ph0123: <p>HI,</p>
<p>I followed the instruction on git.</p>
<ol>
<li>clone code from git.</li>
</ol>
<pre><code class="lang-auto">git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
</code></pre>
<ol start="2">
<li>install dependencies.</li>
</ol>
<pre><code class="lang-auto">pip install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi
</code></pre>
<ol start="3">
<li>run setup and install. The errors were here.<br>
export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/…/"}<br>
python setup.py install</li>
</ol>
<pre><code class="lang-auto">Building wheel torch-1.6.0a0+32bbf12
-- Building version 1.6.0a0+32bbf12
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/cluster/home/cnphuong/pytorch/torch -DCMAKE_PREFIX_PATH=/cluster/software/Anaconda3/2019.03/bin/../ -DNUMPY_INCLUDE_DIR=/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/cluster/home/cnphuong/my_environment/bin/python -DPYTHON_INCLUDE_DIR=/cluster/software/Python/3.6.6-foss-2018b/include/python3.6m -DPYTHON_LIBRARY=/cluster/software/Python/3.6.6-foss-2018b/lib/libpython3.6m.so.1.0 -DTORCH_BUILD_VERSION=1.6.0a0+32bbf12 -DUSE_NUMPY=True /cluster/home/cnphuong/pytorch
CMake Error: The source directory "/cluster/home/cnphuong/pytorch" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
Traceback (most recent call last):
  File "setup.py", line 738, in &lt;module&gt;
    build_deps()
  File "setup.py", line 320, in build_deps
    cmake=cmake)
  File "/cluster/home/cnphuong/pytorch/tools/build_pytorch_libs.py", line 59, in build_caffe2
    rerun_cmake)
  File "/cluster/home/cnphuong/pytorch/tools/setup_helpers/cmake.py", line 324, in generate
    self.run(args, env=my_env)
  File "/cluster/home/cnphuong/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/cluster/software/Python/3.6.6-foss-2018b/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '-GNinja', '-DBUILD_PYTHON=True', '-DBUILD_TEST=True', '-DCMAKE_BUILD_TYPE=Release', '-DCMAKE_INSTALL_PREFIX=/cluster/home/cnphuong/pytorch/torch', '-DCMAKE_PREFIX_PATH=/cluster/software/Anaconda3/2019.03/bin/../', '-DNUMPY_INCLUDE_DIR=/cluster/home/cnphuong/my_environment/lib/python3.6/site-packages/numpy/core/include', '-DPYTHON_EXECUTABLE=/cluster/home/cnphuong/my_environment/bin/python', '-DPYTHON_INCLUDE_DIR=/cluster/software/Python/3.6.6-foss-2018b/include/python3.6m', '-DPYTHON_LIBRARY=/cluster/software/Python/3.6.6-foss-2018b/lib/libpython3.6m.so.1.0', '-DTORCH_BUILD_VERSION=1.6.0a0+32bbf12', '-DUSE_NUMPY=True', '/cluster/home/cnphuong/pytorch']' returned non-zero exit status 1.
</code></pre>
<p>I only want to run with multi CPUs. These step is correct or not?<br>
Thanks,</p>

pritamdamania87: <aside class="quote no-group quote-modified" data-username="ph0123" data-post="3" data-topic="77106">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ph0123/48/22888_2.png" class="avatar"> ph0123:</div>
<blockquote>
<p>CMake Error: The source directory “/cluster/home/cnphuong/pytorch” does not appear to contain CMakeLists.txt.</p>
</blockquote>
</aside>
<p>Can you check if that directory has a CMakeLists.txt file? Usually there should be a CMakeLists.txt file in the top level directory when you clone pytorch.</p>

ph0123: <aside class="quote no-group" data-username="pritamdamania87" data-post="4" data-topic="77106">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<p>y has a CMakeLists.txt file? Usually there should be a CMakeLists.txt file in the top level directory when</p>
</blockquote>
</aside>
<p>Oh. I did not see CMakeLists.txt. I will try to clone again.<br>
Thanks,</p>