Build Error[ninja: build stopped: subcommand failed] when building Pytorch from source

longxd: <p>I have a cuda 9.1 installed, and need to build a pytorch with the newest version (1.4). There is no pre-build binary for that, so I download the source code and try to build it. It fails. It says:</p>
<p>[38/4463] Performing build step for ‘nccl_external’<br>
FAILED: nccl_external-prefix/src/nccl_external-stamp/nccl_external-build nccl/lib/libnccl_static.a<br>
cd /usr/home/work/pytorch10/third_party/nccl/nccl &amp;&amp; env CCACHE_DISABLE=1 SCCACHE_DISABLE=1 make CXX=/usr/local/bin/c++ CUDA_HOME=/usr/local/cuda NVCC=/usr/local/cuda/bin/nvcc NVCC_GENCODE=-gencode=arch=compute_60,code=sm_60 BUILDDIR=/usr/home/work/pytorch10/build/nccl VERBOSE=0 -j &amp;&amp; /usr/home/work/anaconda3/envs/fromsrc1/bin/cmake -E touch /usr/home/work/pytorch10/build/nccl_external-prefix/src/nccl_external-stamp/nccl_external-build<br>
make -C src build BUILDDIR=/usr/home/work/pytorch10/build/nccl</p>
<p>…</p>
<p>Generating rules                               &gt; /usr/home/work/pytorch10/build/nccl/obj/collectives/device/Makefile.rules<br>
/bin/sh: ./gen_rules.sh: /bin/bash^M: bad interpreter: No such file or directory<br>
Compiling  functions.cu                        &gt; /usr/home/work/pytorch10/build/nccl/obj/collectives/device/functions.o<br>
nvlink warning : Function ‘_Z25ncclBroadcastRing_copy_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z27ncclBroadcastRingLL_copy_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z25ncclBroadcastTree_copy_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z27ncclBroadcastTreeLL_copy_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z21ncclReduceRing_sum_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z23ncclReduceRingLL_sum_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z21ncclReduceTree_sum_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z23ncclReduceTreeLL_sum_i8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function ‘_Z21ncclReduceRing_sum_u8P14CollectiveArgs’ has address taken but no possible call to it<br>
nvlink warning : Function '_Z23nccl</p>
<p>…</p>
<p>ninja: build stopped: subcommand failed.<br>
Building wheel torch-1.5.0a0+358ba59<br>
– Building version 1.5.0a0+358ba59<br>
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/usr/home/work/pytorch10/torch -DCMAKE_PREFIX_PATH=/home/work/anaconda3/envs/fromsrc1 -DNUMPY_INCLUDE_DIR=/home/work/anaconda3/envs/fromsrc1/lib/python3.8/site-packages/numpy/core/include -DPYTHON_EXECUTABLE=/home/work/anaconda3/envs/fromsrc1/bin/python -DPYTHON_INCLUDE_DIR=/home/work/anaconda3/envs/fromsrc1/include/python3.8 -DPYTHON_LIBRARY=/home/work/anaconda3/envs/fromsrc1/lib/libpython3.8.so.1.0 -DTORCH_BUILD_VERSION=1.5.0a0+358ba59 -DUSE_NUMPY=True /usr/home/work/pytorch10<br>
cmake --build . --target install --config Release – -j 16<br>
Traceback (most recent call last):<br>
File “setup.py”, line 745, in <br>
build_deps()<br>
File “setup.py”, line 311, in build_deps<br>
build_caffe2(version=version,<br>
File “/usr/home/work/pytorch10/tools/build_pytorch_libs.py”, line 62, in build_caffe2<br>
cmake.build(my_env)<br>
File “/usr/home/work/pytorch10/tools/setup_helpers/cmake.py”, line 339, in build<br>
self.run(build_args, my_env)<br>
File “/usr/home/work/pytorch10/tools/setup_helpers/cmake.py”, line 141, in run<br>
check_call(command, cwd=self.build_dir, env=env)<br>
File “/home/work/anaconda3/envs/fromsrc1/lib/python3.8/subprocess.py”, line 364, in check_call</p>
<p>Can somebody help?</p>

ptrblck: <p>I’m not sure what’s going on, but apparently the 3rd party NCCL fails to install.<br>
Could you install NCCL locally and set <code>USE_SYSTEM_NCCL</code> with its path while compiling?</p>

Epoching: <p>This worked <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<hr>
<p>Anyone trying to install on IBM Power 8/9 machines, I did the following:</p>
<pre><code class="lang-auto">conda install -c conda-forge cudatoolkit nccl cudnn
</code></pre>
<p>Then build from source (Same as <a href="https://github.com/pytorch/pytorch/tree/v1.12.1#from-source" rel="noopener nofollow ugc">documentation</a>, but added the <code>USE_SYSTEM_NCCL=1</code>):</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
USE_SYSTEM_NCCL=1 python setup.py install
</code></pre>
<hr>
<p>This fixed the NCCL errors, but ran into other caffe2 cuda errors afterwards. What worked was upgrading cudatoolkit to be exactly 11.6:</p>
<pre><code class="lang-auto">conda install -c conda-forge cudatoolkit==11.6 nccl cudnn
</code></pre>
<p>Also updating paths ($PATH and $LD_LIBRARY_PATH variables) to use <code>gcc 8.3.1</code> and <code>cuda 11.6</code></p>
<p>Then running commands exactly as they are from documentation worked fine:</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
USE_SYSTEM_NCCL=1 python setup.py install
</code></pre>
<hr>