Can not build pytorch eiether in ubuntu or centos

GGPYTORCH000: <p>I can never build it according to instruction: <a href="https://github.com/pytorch/pytorch#from-source" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/pytorch: Tensors and Dynamic neural networks in Python with strong GPU acceleration</a><br>
For ubuntu 18.04 here is the last few lines of build failure (log at bottom is Centos 8 build failure as well)</p>
<pre><code class="lang-auto">
Get:2 http://ppa.launchpad.net/deadsnakes/ppa/ubuntu bionic/main amd64 python3.9-distutils all 3.9.11-1+bionic1 [190 kB]
Fetched 315 kB in 2s (194 kB/s)              
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package python3.9-lib2to3.
(Reading database ... 14757 files and directories currently installed.)
Preparing to unpack .../python3.9-lib2to3_3.9.11-1+bionic1_all.deb ...
Unpacking python3.9-lib2to3 (3.9.11-1+bionic1) ...
Selecting previously unselected package python3.9-distutils.
Preparing to unpack .../python3.9-distutils_3.9.11-1+bionic1_all.deb ...
Unpacking python3.9-distutils (3.9.11-1+bionic1) ...
Setting up python3.9-lib2to3 (3.9.11-1+bionic1) ...
Setting up python3.9-distutils (3.9.11-1+bionic1) ...
root@ixt-hq-178:/pytorch# python3 setup.py install
Traceback (most recent call last):
  File "/pytorch/setup.py", line 219, in &lt;module&gt;
    from setuptools import setup, Extension, find_packages
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 14, in &lt;module&gt;
    from setuptools.dist import Distribution, Feature
  File "/usr/lib/python3/dist-packages/setuptools/dist.py", line 24, in &lt;module&gt;
    from setuptools.depends import Require
  File "/usr/lib/python3/dist-packages/setuptools/depends.py", line 7, in &lt;module&gt;
    from .py33compat import Bytecode
  File "/usr/lib/python3/dist-packages/setuptools/py33compat.py", line 54, in &lt;module&gt;
    unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
AttributeError: 'HTMLParser' object has no attribute 'unescape'
root@ixt-hq-178:/pytorch# sudo apt install python3-distutils
sudo: unable to resolve host ixt-hq-178
Reading package lists... Done
Building dependency tree       
Reading state information... Done
python3-distutils is already the newest version (3.6.9-1~18.04).
python3-distutils set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@ixt-hq-178:/pytorch#
</code></pre>
<p>For centOS 8.x:</p>
<pre><code class="lang-auto">
[root@centos pytorch]# !123
python setup.py build --cmake-only
Traceback (most recent call last):
File "setup.py", line 219, in
Traceback (most recent call last):
File "setup.py", line 219, in
from setuptools import setup, Extension, find_packages
File "/usr/local/lib/python3.8/site-packages/setuptools/**init** .py", line 18, in
from setuptools.dist import Distribution
File "/usr/local/lib/python3.8/site-packages/setuptools/dist.py", line 34, in
from setuptools import windows_support
File "/usr/local/lib/python3.8/site-packages/setuptools/windows_support.py", line 2, in
import ctypes
File "/usr/local/lib/python3.8/ctypes/**init** .py", line 7, in
from _ctypes import Union, Structure, Array
ModuleNotFoundError: No module named '_ctypes'
</code></pre>

ptrblck: <p>In both cases it seems you are running into issues using older Python packages.<br>
E.g. searching for the first issue, yields <a href="https://stackoverflow.com/a/65640477">this result</a>.</p>

GGPYTORCH000: <p>no i did not help, already tried it and it fails.</p>
<pre><code class="lang-auto">
/root/pt/Python-3.9.10/pytorch/torch/nn/functional.pyi.in -&gt; /root/pt/Python-3.9.10/pytorch/torch/nn/functional.pyi.in skipped
/root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/timeit_template.cpp -&gt; /root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/timeit_template.cpp skipped
/root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/compat_bindings.cpp -&gt; /root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/compat_bindings.cpp skipped
/root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/timer_callgrind_template.cpp -&gt; /root/pt/Python-3.9.10/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/timer_callgrind_template.cpp skipped
/root/pt/Python-3.9.10/pytorch/torch/utils/data/datapipes/datapipe.pyi.in -&gt; /root/pt/Python-3.9.10/pytorch/torch/utils/data/datapipes/datapipe.pyi.in skipped
Traceback (most recent call last):
  File "/root/pt/Python-3.9.10/pytorch/setup.py", line 424, in check_pydep
Building wheel torch-1.12.0a0+git23383b1
-- Building version 1.12.0a0+git23383b1
    importlib.import_module(importname)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "&lt;frozen importlib._bootstrap&gt;", line 1030, in _gcd_import
  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'yaml'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/pt/Python-3.9.10/pytorch/setup.py", line 906, in &lt;module&gt;
    build_deps()
  File "/root/pt/Python-3.9.10/pytorch/setup.py", line 370, in build_deps
    check_pydep('yaml', 'pyyaml')
  File "/root/pt/Python-3.9.10/pytorch/setup.py", line 426, in check_pydep
    raise RuntimeError(missing_pydep.format(importname=importname, module=module))
RuntimeError: Missing build dependency: Unable to `import yaml`.
Please install it via `conda install pyyaml` or `pip install pyyaml`
</code></pre>

ptrblck: <p>The current issue points to:</p>
<pre><code class="lang-python">ModuleNotFoundError: No module named 'yaml'
...
Please install it via `conda install pyyaml` or `pip install pyyaml`
</code></pre>
<p>so did you try to install the missing package?</p>

GGPYTORCH000: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>pip install pyyaml</code></p>
</blockquote>
</aside>
<p>This is already installed. That is why i am questioning here, why it is asking for packages that is already installed.</p>
<p>[root@slurm-0 /]# pip install pyyaml<br>
Collecting pyyaml<br>
Downloading PyYAML-6.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (661 kB)<br>
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 661.8/661.8 KB 9.8 MB/s eta 0:00:00<br>
Installing collected packages: pyyaml<br>
Successfully installed pyyaml-6.0<br>
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <a href="https://pip.pypa.io/warnings/venv" class="inline-onebox" rel="noopener nofollow ugc">12. Virtual Environments and Packages — Python 3.12.0 documentation</a><br>
[root@slurm-0 /]# pip3 install pyyaml<br>
Requirement already satisfied: pyyaml in /usr/local/lib/python3.9/site-packages (6.0)<br>
WARNING: Running pip as the ‘root’ user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: <a href="https://pip.pypa.io/warnings/venv" class="inline-onebox" rel="noopener nofollow ugc">12. Virtual Environments and Packages — Python 3.12.0 documentation</a><br>
[root@slurm-0 /]#</p>

GGPYTORCH000: <p>Actually this time, somehow it moves forward with build. Now I am getting following:</p>
<p>(last lines of build)</p>
<p>/root/pt/pytorch/torch/csrc/utils/variadic.cpp → /root/pt/pytorch/torch/csrc/utils/variadic.cpp skipped<br>
/root/pt/pytorch/torch/csrc/utils/variadic.h → /root/pt/pytorch/torch/csrc/utils/variadic.h skipped<br>
/root/pt/pytorch/torch/lib/libshm/alloc_info.h → /root/pt/pytorch/torch/lib/libshm/alloc_info.h skipped<br>
/root/pt/pytorch/torch/lib/libshm/core.cpp → /root/pt/pytorch/torch/lib/libshm/core.cpp skipped<br>
/root/pt/pytorch/torch/lib/libshm/err.h → /root/pt/pytorch/torch/lib/libshm/err.h skipped<br>
/root/pt/pytorch/torch/lib/libshm/libshm.h → /root/pt/pytorch/torch/lib/libshm/libshm.h skipped<br>
/root/pt/pytorch/torch/lib/libshm/manager.cpp → /root/pt/pytorch/torch/lib/libshm/manager.cpp skipped<br>
/root/pt/pytorch/torch/lib/libshm/socket.h → /root/pt/pytorch/torch/lib/libshm/socket.h skipped<br>
/root/pt/pytorch/torch/lib/libshm_windows/core.cpp → /root/pt/pytorch/torch/lib/libshm_windows/core.cpp skipped<br>
/root/pt/pytorch/torch/lib/libshm_windows/libshm.h → /root/pt/pytorch/torch/lib/libshm_windows/libshm.h skipped<br>
/root/pt/pytorch/torch/nn/functional.pyi.in → /root/pt/pytorch/torch/nn/functional.pyi.in skipped<br>
/root/pt/pytorch/torch/utils/benchmark/utils/timeit_template.cpp → /root/pt/pytorch/torch/utils/benchmark/utils/timeit_template.cpp skipped<br>
/root/pt/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/compat_bindings.cpp → /root/pt/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/compat_bindings.cpp skipped<br>
/root/pt/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/timer_callgrind_template.cpp → /root/pt/pytorch/torch/utils/benchmark/utils/valgrind_wrapper/timer_callgrind_template.cpp skipped<br>
/root/pt/pytorch/torch/utils/data/datapipes/datapipe.pyi.in → /root/pt/pytorch/torch/utils/data/datapipes/datapipe.pyi.in skipped<br>
gmake: Makefile: No such file or directory<br>
gmake: *** No rule to make target ‘Makefile’.  Stop.<br>
Building wheel torch-1.12.0a0+gitebeea9e<br>
– Building version 1.12.0a0+gitebeea9e<br>
cmake3 --build . --target install --config Release – -j 56</p>
<p>This is when building with amdgpu<br>
PYTORCH_ROCM_ARCH=gfx90? python setup.py install</p>

ptrblck: <aside class="quote no-group" data-username="GGPYTORCH000" data-post="6" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggpytorch000/48/47787_2.png" class="avatar"> GGPYTORCH000:</div>
<blockquote>
<p>This is when building with amdgpu<br>
PYTORCH_ROCM_ARCH=gfx90? python setup.py install</p>
</blockquote>
</aside>
<p>Sorry, I’m not deeply familiar with AMD builds, so let’s wait for an AMD expert <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

GGPYTORCH000: <p>on platform with nvidia gpu, it fails also spectacularly with following error:</p>
<pre><code class="lang-auto">root@nonroot-MS-7B22:/build-scripts/pt/pytorch# python setup.py install  | tee build-pt.log
CMake Error: File /usr/local/bin/cmake/Modules/CMakeSystem.cmake.in does not exist.
CMake Error at /usr/local/share/cmake-3.16/Modules/CMakeDetermineSystem.cmake:185 (configure_file):
  configure_file Problem configuring file
Call Stack (most recent call first):
  CMakeLists.txt:27 (project)


CMake Error at /usr/local/share/cmake-3.16/Modules/CMakeDetermineCXXCompiler.cmake:23 (include):
  include could not find load file:

    /usr/local/bin/cmake/Modules/CMakeDetermineCompiler.cmake
Call Stack (most recent call first):
  CMakeLists.txt:27 (project)


CMake Error at /usr/local/share/cmake-3.16/Modules/CMakeDetermineCXXCompiler.cmake:64 (_cmake_find_compiler):
  Unknown CMake command "_cmake_find_compiler".
Call Stack (most recent call first):
  CMakeLists.txt:27 (project)


CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
See also "/build-scripts/pt/pytorch/build/CMakeFiles/CMakeOutput.log".
Building wheel torch-1.12.0a0+git5375b2e
-- Building version 1.12.0a0+git5375b2e
cmake -GNinja -DBUILD_PYTHON=True -DBUILD_TEST=True -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/build-scripts/pt/pytorch/torch -DCMAKE_PREFIX_PATH=/usr/local/lib/python3.9/site-packages;/anaconda3 -DCMAKE_ROOT=/usr/local/bin/cmake -DPYTHON_EXECUTABLE=/usr/bin/python -DPYTHON_INCLUDE_DIR=/usr/local/include/python3.9 -DPYTHON_LIBRARY=/usr/local/lib/libpython3.9.a -DTORCH_BUILD_VERSION=1.12.0a0+git5375b2e -DUSE_NUMPY=True /build-scripts/pt/pytorch
root@nonroot-MS-7B22:/build-scripts/pt/pytorch#
</code></pre>

ptrblck: <aside class="quote no-group" data-username="GGPYTORCH000" data-post="8" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggpytorch000/48/47787_2.png" class="avatar"> GGPYTORCH000:</div>
<blockquote>
<p><code>CMake Error: File /usr/local/bin/cmake/Modules/CMakeSystem.cmake.in does not exist.</code></p>
</blockquote>
</aside>
<p>I don’t know what’s causing this, but <code>cmake</code> ships with this file in <a href="https://github.com/Kitware/CMake/blob/843d1d198d15f65781d71118573461fb03a3101c/Modules/CMakeSystem.cmake.in">this location</a> so it seems you <code>cmake</code> setup might be broken?</p>

GGPYTORCH000: <p>naah, other builds using cmake working just fine, so with datapoints i have, i am going to point finger at pytorch itself is broken than cmake. after all i followed exactly what is instruction says under pytorch build instruction which itself seems incomplete at best.</p>

tk0320: <p>i encountered this problem as well.<br>
error message prompts the file cmake did not find is located at differ from cmake bin running location. then, i found out setup.py define a CMAKE_ROOT env to a unexpected dir. inspecting building process, it tells script inherit the envs from your system envs. finally, i find out CMAKE_ROOT defined in .bashrc, which i copied from another development machine. that’s why the error caused and the bug is really rare.</p>

GGPYTORCH000: <p>i am not sure if responder is from meta, however he attempts to shift problem to GPU vendor and then later on cmake randomly and then appears to have bailed out when that logic is shot down. I dont think pytorch ever built successfully or reliably based on the instruction provided publiclt.</p>

ptrblck: <aside class="quote no-group" data-username="GGPYTORCH000" data-post="12" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggpytorch000/48/47787_2.png" class="avatar"> GGPYTORCH000:</div>
<blockquote>
<p>however he attempts to shift problem to GPU vendor</p>
</blockquote>
</aside>
<p>Understandably, I won’t comment on build systems I’ve never used.</p>
<aside class="quote no-group" data-username="GGPYTORCH000" data-post="12" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggpytorch000/48/47787_2.png" class="avatar"> GGPYTORCH000:</div>
<blockquote>
<p>and then later on cmake randomly and then appears to have bailed out when that logic is shot down</p>
</blockquote>
</aside>
<p>Unfortunately, you did not follow up with any constructive feedback or more information besides pointing fingers at PyTorch. Unsure what your expectation is since I can build from source and did not see your error before.</p>
<aside class="quote no-group" data-username="GGPYTORCH000" data-post="12" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ggpytorch000/48/47787_2.png" class="avatar"> GGPYTORCH000:</div>
<blockquote>
<p>I dont think pytorch ever built successfully or reliably based on the instruction provided publiclt.</p>
</blockquote>
</aside>
<p>I will happily point to our nightly builds using these instructions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="tk0320" data-post="11" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/tk0320/48/14342_2.png" class="avatar"> tk0320:</div>
<blockquote>
<p>i found out setup.py define a CMAKE_ROOT env to a unexpected dir. inspecting building process, it tells script inherit the envs from your system envs. finally, i find out CMAKE_ROOT defined in .bashrc, which i copied from another development machine. that’s why the error caused and the bug is really rare.</p>
</blockquote>
</aside>
<p>That’s a great explanation and thanks for following up! This would explain why <code>cmake</code> could not find the file I’ve pointed out was shipped in its package.</p>

tk0320: <p>altough, i execute the building process different from you, but i think the problem is the same.</p>
<p><code>-DCMAKE_ROOT=/usr/local/bin/cmake</code> results in  searching into path likes<code>/usr/local/bin/cmake/Modules/...</code>. however, as you see, <code>/usr/local/share/cmake-3.16/Modules/</code> is the corrected path.</p>

GGPYTORCH000: <aside class="quote no-group" data-username="ptrblck" data-post="13" data-topic="147057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I will happily point to our nightly builds using these instructions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>i dont think this will solve to anything → “I will happily point to our nightly builds using these instructions. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20">”</p>
<p>has been down on this path before, will work for timebeing only to see somethign else broken. I am not interested in being QA job for pytorch build issues.</p>
<p>My better suggestion is that build documentation is improved an if CI is running, failed builds are reduced . It is not only the error i have seen when I attempted numeriosu times to build torch, and failed with all sort of issues.<br>
This compared to many open source project that i worked that builds flawlessly.</p>