Error install from source on server

username: ph0123
cooked: <p>HI,<br>
I follow instructions from pytorch git as below.</p>
<blockquote>
<ol>
<li>
<p>install dependencies<br>
conda install numpy ninja pyyaml mkl mkl-include setuptools cmake cffi<br>
(cmake error: conda install -c anaconda cmake)</p>
</li>
<li>
<p>clone pytorch<br>
git clone --recursive <a href="https://github.com/pytorch/pytorch" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
cd pytorch</p>
</li>
<li>
<p>export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-“$(dirname $(which conda))/…/”}<br>
python setup.py install</p>
</li>
</ol>
</blockquote>
<p>The error is as below.</p>
<blockquote>
<p>MakeFiles/c10.dir/util/numa.cpp.o -c …/c10/util/numa.cpp<br>
…/c10/util/numa.cpp:6:10: fatal error: numa.h: No such file or directory<br>
<span class="hashtag-raw">#include</span> &lt;numa.h&gt;<br>
^~~~~~~~<br>
compilation terminated.<br>
[1684/4092] Building CXX object third_p…akeFiles/dnnl_cpu.dir/cpu_reorder.cpp.o<br>
ninja: build stopped: subcommand failed.<br>
Traceback (most recent call last):<br>
File “setup.py”, line 740, in <br>
build_deps()<br>
File “setup.py”, line 323, in build_deps<br>
cmake=cmake)<br>
File “/cluster/home/cnphuong/pytorch/tools/build_pytorch_libs.py”, line 62, in build_caffe2<br>
cmake.build(my_env)<br>
File “/cluster/home/cnphuong/pytorch/tools/setup_helpers/cmake.py”, line 340, in build<br>
self.run(build_args, my_env)<br>
File “/cluster/home/cnphuong/pytorch/tools/setup_helpers/cmake.py”, line 141, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/cluster/home/cnphuong/.conda/envs/Pytorch_ENV/lib/python3.6/subprocess.py”, line 311, in check_call<br>
raise CalledProcessError(retcode, cmd)<br>
subprocess.CalledProcessError: Command ‘[‘cmake’, ‘–build’, ‘.’, ‘–target’, ‘install’, ‘–config’, ‘Release’, ‘–’, ‘-j’, ‘64’]’ returned non-zero exit status 1.</p>
</blockquote>
<p>When run setup.py I saw some output <a href="https://drive.google.com/open?id=1QzKKK6Ul4o9b37bIVf_x0byjV0FYYjl7" rel="noopener nofollow ugc">here</a>.</p>

username: mrshenli
cooked: <p>Could you please create an issue on github to track this?</p>
<p>And this does not seem to be relevant to <code>torch.diistributed</code>?</p>

username: ph0123
cooked: <p>hi,<br>
Because Distributed CPUs is only supported by building from source. I think most of people with this tag have more experiments than others.</p>
<p>Thanks,</p>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/ph0123">@ph0123</a>, <code>DistributedDataParallel</code> with CPU model should be supported by default in the release binaries. You can enable this mode by passing in a CPU model and do <strong>not</strong> provide a <code>device_ids</code> argument. If this is all you need, you don’t need to compile from source I think? Did you hit any error when trying to run DDP with CPU models?</p>

username: ph0123
cooked: <aside class="quote no-group" data-username="mrshenli" data-post="4" data-topic="78876">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mrshenli/48/12220_2.png" class="avatar"> mrshenli:</div>
<blockquote>
<p><code>tedDataParallel</code> with CPU model should be supported by default in the release binaries. You can enable this mode by passing in a CPU model and do <strong>not</strong> provide a <code>device_ids</code> argument. If this is all you need, you don’t need to compile from source I think? Did you hit any error when trying to run DDP with CPU models</p>
</blockquote>
</aside>
<p>Dear mrshenli,</p>
<p>Thanks, last time, I install from sources, and some errors when run the program with Distributed Parallel. I check the tutorial, and I have to install from source to run Distributed CPUs.</p>
<p>But now I read the documents again, it do not need install from sources.</p>
<p>I will try. Thank you so much!<br>
Thanks,</p>

username: ph0123
cooked: <p>Hi,<br>
Please see</p>
<pre><code class="lang-auto">**MPI Backend**

The Message Passing Interface (MPI) is a standardized tool from the field of high-performance computing. It allows to do point-to-point and collective communications and was the main inspiration for the API of `torch.distributed` . Several implementations of MPI exist (e.g. [Open-MPI](https://www.open-mpi.org/), [MVAPICH2](http://mvapich.cse.ohio-state.edu/), [Intel MPI](https://software.intel.com/en-us/intel-mpi-library)) each optimized for different purposes. The advantage of using the MPI backend lies in MPI’s wide availability - and high-level of optimization - on large computer clusters. [Some](https://developer.nvidia.com/mvapich) [recent](https://developer.nvidia.com/ibm-spectrum-mpi) [implementations](https://www.open-mpi.org/) are also able to take advantage of CUDA IPC and GPU Direct technologies in order to avoid memory copies through the CPU.

Unfortunately, PyTorch’s binaries can not include an MPI implementation and we’ll have to recompile it by hand. Fortunately, this process is fairly simple given that upon compilation, PyTorch will look *by itself* for an available MPI implementation. The following steps install the MPI backend, by installing PyTorch [from source](https://github.com/pytorch/pytorch#from-source).

1. Create and activate your Anaconda environment, install all the pre-requisites following [the guide](https://github.com/pytorch/pytorch#from-source), but do **not** run `python setup.py install` yet.
2. Choose and install your favorite MPI implementation. Note that enabling CUDA-aware MPI might require some additional steps. In our case, we’ll stick to Open-MPI *without* GPU support: `conda install -c conda-forge openmpi`
3. Now, go to your cloned PyTorch repo and execute `python setup.py install` .

In order to test our newly installed backend, a few modifications are required.

1. Replace the content under `if __name__ == '__main__':` with `init_process(0, 0, run, backend='mpi')` .
2. Run `mpirun -n 4 python myscript.py` .

The reason for these changes is that MPI needs to create its own environment before spawning the processes. MPI will also spawn its own processes and perform the handshake described in [Initialization Methods](https://pytorch.org/tutorials/intermediate/dist_tuto.html#initialization-methods), making the `rank` and `size` arguments of `init_process_group` superfluous. This is actually quite powerful as you can pass additional arguments to `mpirun` in order to tailor computational resources for each process. (Things like number of cores per process, hand-assigning machines to specific ranks, and [some more](https://www.open-mpi.org/faq/?category=running#mpirun-hostfile)) Doing so, you should obtain the same familiar output as with the other communication backends.
</code></pre>
<p>Thanks,</p>

username: mrshenli
cooked: <p>Oh I see, you are trying to use MPI. Is MPI the only option, or will Gloo or NCCL also be acceptable?</p>
<p>And yes, MPI backend needs building from source.</p>

username: mrshenli
cooked: <p>BTW, the build log shown <a href="https://drive.google.com/file/d/1QzKKK6Ul4o9b37bIVf_x0byjV0FYYjl7/view" rel="nofollow noopener">here</a> does not seem to be complete. Could you please also paste the last few screens of logs?</p>

username: ph0123
cooked: <p>Hi,<br>
I try to work with MPI backend but it did not work.<br>
I change to Gloo backend.<br>
Thanks,</p>