Ninja error when installing PyTorch

username: seankala
cooked: <p>Hello. I’m trying to install PyTorch v1.4.0 from source and am experiencing the following error:</p>
<pre><code class="lang-auto">[4070/4411] Building CXX object test_api/CMakeFiles/test_api.dir/modules.cpp.o
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "setup.py", line 743, in &lt;module&gt;
    build_deps()
  File "setup.py", line 316, in build_deps
    cmake=cmake)
  File "/home/seokwon/pytorch/tools/build_pytorch_libs.py", line 62, in build_caffe2
    cmake.build(my_env)
  File "/home/seokwon/pytorch/tools/setup_helpers/cmake.py", line 339, in build
    self.run(build_args, my_env)
  File "/home/seokwon/pytorch/tools/setup_helpers/cmake.py", line 141, in run
    check_call(command, cwd=self.build_dir, env=env)
  File "/home/seokwon/anaconda3/envs/seanconda/lib/python3.7/subprocess.py", line 363, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['cmake', '--build', '.', '--target', 'install', '--config', 'Release', '--', '-j', '20']' returned non-zero exit status 1.
</code></pre>
<p>I followed all of the steps accordingly specified in the GitHub README file and also made sure that CUDA version was appropriate. Does anybody know how to solve this issue? I’ve tried looking elsewhere online but was unsuccessful. Thanks in advance!</p>

username: ptrblck
cooked: <p>The error message is not very helpful. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
Could you check, if you are running out of memory?</p>

username: seankala
cooked: <p>Sorry, but how do I check if I’m running out of memory?</p>
<p>Also on a side note, the server I’m using has two versions of CUDA installed: 9.0 and 10.1. In the <code>/usr/local/</code> directory there are both <code>cuda-10.1/</code> and <code>cuda-9.0/</code>, and the <code>cuda</code> file is pointing to <code>/usr/local/cuda-9.0/</code> (I’m not sure if this is correct terminology). I’m trying to use CUDA 10.1, would this be the cause of the problem? If I install PyTorch using <code>conda install pytorch torchvision cudatoolkit=10.1 -c pytorch</code> then it install fine, but <code>torch.cuda.is_available()</code> returns <code>False</code>.</p>

username: seankala
cooked: <p>So I managed to solve my issue, but I haven’t been able to solve this error per se.</p>
<p>I wasn’t aware that the CUDA driver didn’t match CUDA. I assumed that my server administrator took care of it but perhaps there was an error in the process. Regardless, installing PyTorch with the method specified on PyTorch’s homepage now works.</p>

username: ptrblck
cooked: <p>Good to hear you figured out the (binary?) installation issue.</p>
<p>However, that doesn’t solve the build from source failure.<br>
Could you clean the build (<code>python setup.py clean</code>), update all submodules (<code>git submodule update --init --recursive</code>) and rebuild it?</p>
<p>While the build is running, you could use <code>htop</code> to observe the system RAM.</p>

username: seankala
cooked: <p>I made a new virtual environment and did as you suggested, but it doesn’t seem like it’s a memory problem. There’s plenty of memory left even when the process is taking place.</p>

username: ptrblck
cooked: <p>Is it always throwing this error at the same step and did a clean build not solve anything?</p>

username: seankala
cooked: <p>It’s not always at the same time step (e.g. this time it stopped at <code>[3804/4411]</code>) but the error message is the same.</p>

username: ptrblck
cooked: <p>Do you see any other errors before this step in the log?<br>
Since the build uses multiple workers, the actual error might be further up in the log.</p>

username: Yinglei_Zhang
cooked: <p>I am facing the same issue when build PyTorch-1.4.0 from source.</p>
<p>Here is the building log: <a href="https://gist.github.com/YingleiZhang/fc98c94d493ac39506daacefa0298b38#file-gistfile1-txt-L11" rel="nofollow noopener">https://gist.github.com/YingleiZhang/fc98c94d493ac39506daacefa0298b38#file-gistfile1-txt-L11</a></p>
<p>I don’t have this problem when build 1.2.0. Any help is appreciated.</p>

username: thomasB
cooked: <p>I got the same errors when installing pytorch from source for versions 1.6 and 1.7 on Ubuntu 18.04. I installed the nvidia drivers, cuda 10.2 and cudnn 8.0.3 for 2080ti gpus. I followed the instructions to install pytorch from source and still got the ninja error from above also at “random” points of the installation. I didn’t have these problems on my local machine with Ubuntu 20.04 with the same GPU and same versions. Any help is much appreciated.</p>

username: ptrblck
cooked: <p>The previous error was created by a potentially old version of <code>GLIBCXX</code>:</p>
<pre><code class="lang-auto">../../lib/libc10.so: error: undefined reference to 'std::thread::_State::~_State()', version 'GLIBCXX_3.4.22'
../../lib/libc10.so: error: undefined reference to 'typeinfo for std::thread::_State', version 'GLIBCXX_3.4.22'
../../lib/libc10.so: error: undefined reference to 'std::runtime_error::runtime_error(char const*)', version 'GLIBCXX_3.4.21'
</code></pre>
<p>Could you check the install log for errors?<br>
Note that the installation might continue for some steps and the error might be shown in previous lines.</p>

username: sumantrom
cooked: <p>I’ve proposed a solution, it will be great if people can go ahead and do the following changes.</p>
<p>Add <span class="hashtag-raw">#include</span>  in the following files<br>
pytorch/third_party/tensorpipe/tensorpipe/common/allocator.h<br>
pytorch/third_party/tensorpipe/tensorpipe/common/memory.h<br>
pytorch/third_party/gloo/gloo/types.h</p>
<p>In fedora 42, site package 3.13 python ideally shouldn’t be played with. I have gone ahead and added a few lines to take care of that in <a href="https://github.com/pytorch/pytorch/pull/154812" class="inline-onebox" rel="noopener nofollow ugc">add fixes for missing headers, setup.py for local build by sumantro93 · Pull Request #154812 · pytorch/pytorch · GitHub</a></p>
<p>Note this is on Fedora 42 + CPU my final build successful output is</p>
<pre><code class="lang-auto"> ~/pytorch  main !1  python setup.py develop                                                                                                                    1 err  torch py  11:59:04 
Building wheel torch-2.8.0a0+git5616fa4
-- Building version 2.8.0a0+git5616fa4
-- Checkout nccl release tag: v2.26.5-1
cmake --build . --target install --config Release
[1/2] Install the project...
-- Install configuration: "Release"
running develop
/usr/lib/python3.13/site-packages/setuptools/command/develop.py:41: EasyInstallDeprecationWarning: easy_install command is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` and ``easy_install``.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://github.com/pypa/setuptools/issues/917 for details.
        ********************************************************************************

!!
  easy_install.initialize_options(self)
/usr/lib/python3.13/site-packages/setuptools/_distutils/cmd.py:66: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running egg_info
writing torch.egg-info/PKG-INFO
writing dependency_links to torch.egg-info/dependency_links.txt
writing entry points to torch.egg-info/entry_points.txt
writing requirements to torch.egg-info/requires.txt
writing top-level names to torch.egg-info/top_level.txt
reading manifest file 'torch.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.*' under directory 'ios'
warning: no files found matching '*.*' under directory 'modules'
warning: no previously-included files matching '*.o' found anywhere in distribution
warning: no previously-included files matching '*.dylib' found anywhere in distribution
warning: no previously-included files matching '*.swp' found anywhere in distribution
adding license file 'LICENSE'
adding license file 'NOTICE'
writing manifest file 'torch.egg-info/SOURCES.txt'
running build_ext
-- Building with NumPy bindings
-- Not using cuDNN
-- Not using CUDA
-- Not using XPU
-- Using MKLDNN
-- Not using Compute Library for the Arm architecture with MKLDNN
-- Not using CBLAS in MKLDNN
-- Not using NCCL
-- Building with distributed package: 
  -- USE_TENSORPIPE=True
  -- USE_GLOO=True
  -- USE_MPI=False
-- Building Executorch
-- Using ITT
Copying functorch._C from functorch/functorch.so to /home/sumantro/pytorch/build/lib.linux-x86_64-cpython-313/functorch/_C.cpython-313-x86_64-linux-gnu.so
copying functorch/functorch.so -&gt; /home/sumantro/pytorch/build/lib.linux-x86_64-cpython-313/functorch/_C.cpython-313-x86_64-linux-gnu.so
building 'torch._C' extension
creating build/temp.linux-x86_64-cpython-313
creating build/temp.linux-x86_64-cpython-313/torch
creating build/temp.linux-x86_64-cpython-313/torch/csrc
gcc -fno-strict-overflow -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -fexceptions -fcf-protection -fexceptions -fcf-protection -fexceptions -fcf-protection -O3 -fPIC -I/usr/include/python3.13 -c torch/csrc/stub.c -o build/temp.linux-x86_64-cpython-313/torch/csrc/stub.o -Wall -Wextra -Wno-strict-overflow -Wno-unused-parameter -Wno-missing-field-initializers -Wno-unknown-pragmas -fno-strict-aliasing
creating build/lib.linux-x86_64-cpython-313/torch
gcc -shared build/temp.linux-x86_64-cpython-313/torch/csrc/stub.o -L/home/sumantro/pytorch/torch/lib -L/usr/lib64 -ltorch_python -o build/lib.linux-x86_64-cpython-313/torch/_C.cpython-313-x86_64-linux-gnu.so -Wl,-rpath,$ORIGIN/lib
copying build/lib.linux-x86_64-cpython-313/torch/_C.cpython-313-x86_64-linux-gnu.so -&gt; torch
Creating /home/sumantro/.local/lib/python3.13/site-packages/torch.egg-link (link to .)
Adding torch 2.8.0a0+git5616fa4 to easy-install.pth file
Installing torchfrtrace script to /home/sumantro/.local/bin
Installing torchrun script to /home/sumantro/.local/bin

Installed /home/sumantro/pytorch
Processing dependencies for torch==2.8.0a0+git5616fa4
Searching for setuptools==74.1.3
Best match: setuptools 74.1.3
Adding setuptools 74.1.3 to easy-install.pth file

Using /usr/lib/python3.13/site-packages
Searching for fsspec==2024.6.1
Best match: fsspec 2024.6.1
Adding fsspec 2024.6.1 to easy-install.pth file

Using /home/sumantro/.local/lib/python3.13/site-packages
Searching for jinja2==3.1.6
Best match: jinja2 3.1.6
Adding jinja2 3.1.6 to easy-install.pth file

Using /usr/lib/python3.13/site-packages
Searching for networkx==3.3
Best match: networkx 3.3
Adding networkx 3.3 to easy-install.pth file

Using /home/sumantro/.local/lib/python3.13/site-packages
Searching for sympy==1.14.0
Best match: sympy 1.14.0
Adding sympy 1.14.0 to easy-install.pth file
Installing isympy script to /home/sumantro/.local/bin

Using /home/sumantro/.local/lib/python3.13/site-packages
Searching for typing-extensions==4.12.2
Best match: typing-extensions 4.12.2
Adding typing-extensions 4.12.2 to easy-install.pth file

Using /usr/lib/python3.13/site-packages/setuptools/_vendor
Searching for filelock==3.15.4
Best match: filelock 3.15.4
Adding filelock 3.15.4 to easy-install.pth file

Using /usr/lib/python3.13/site-packages
Searching for MarkupSafe==3.0.2
Best match: MarkupSafe 3.0.2
Adding MarkupSafe 3.0.2 to easy-install.pth file

Using /usr/lib64/python3.13/site-packages
Searching for mpmath==1.3.0
Best match: mpmath 1.3.0
Adding mpmath 1.3.0 to easy-install.pth file

Using /home/sumantro/.local/lib/python3.13/site-packages
Finished processing dependencies for torch==2.8.0a0+git5616fa4
</code></pre>