How to configure cmake while building pytorch from source code?

username: studyself
cooked: <p>HI,<br>
I need to use FBEGMM in pytorch, and I’m building pytorch from source code.<br>
The errors ask me to set cmake configuration as<br>
-DENABLE_PRECOMPILED_HEADERS=OFF.</p>
<p>I modify cmake.py and try to add -DENABLE_PRECOMPILED_HEADERS=OFF into build_options as<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39815228695790a0e5af2c57af55442e5fd9bc15.png" data-download-href="https://discuss.pytorch.org/uploads/default/39815228695790a0e5af2c57af55442e5fd9bc15" title="aff4f0a091a4086ed9db59aec333eb3"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/9/39815228695790a0e5af2c57af55442e5fd9bc15.png" alt="aff4f0a091a4086ed9db59aec333eb3" data-base62-sha1="8cIhE6VjWmDUDiVl806514RyEwR" width="690" height="394" data-dominant-color="F5F3F1"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">aff4f0a091a4086ed9db59aec333eb3</span><span class="informations">730×417 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>However, the building process prompts:<br>
CMake Warning:<br>
Manually-specified variables were not used by the project:</p>
<pre><code class="lang-auto">CMAKE_CFLAGS_ISYSTEM
ENABLE_PRECOMPILED_HEADERS
</code></pre>
<p>/<em><strong><strong><strong><strong><strong><strong><strong><strong>Below is Errors from log</strong></strong></strong></strong></strong></strong></strong></strong></em>********** ***********/</p>
<pre><code class="lang-auto">[3991/6869] Building CXX object third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o
FAILED: third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o
/usr/bin/c++ -I/qssd/ptSRC/cmake/…/third_party/benchmark/include -I/qssd/ptSRC/cmake/…/third_party/cudnn_frontend/include -I/qssd/ptSRC/third_party/tensorpipe -I/qssd/ptSRC/build/third_party/tensorpipe -I/qssd/ptSRC/third_party/tensorpipe/third_party/libnop/include -I/qssd/ptSRC/third_party/tensorpipe/third_party/libuv/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googlemock/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googletest/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /opt/intel/mkl/include -isystem /qssd/ptSRC/third_party/gemmlowp -isystem /qssd/ptSRC/third_party/neon2sse -isystem /qssd/ptSRC/third_party/XNNPACK/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/…/third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -O3 -DNDEBUG -fPIC -std=gnu++14 -MD -MT third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o -MF third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o.d -o third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/helpers.cc.o -c /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.cc
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.h:13,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/helpers.cc:9:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next &lt;stdlib.h&gt;
^~~~~~~~~~
compilation terminated.
[3993/6869] Building CXX object third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o
FAILED: third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o
/usr/bin/c++ -I/qssd/ptSRC/cmake/…/third_party/benchmark/include -I/qssd/ptSRC/cmake/…/third_party/cudnn_frontend/include -I/qssd/ptSRC/third_party/tensorpipe -I/qssd/ptSRC/build/third_party/tensorpipe -I/qssd/ptSRC/third_party/tensorpipe/third_party/libnop/include -I/qssd/ptSRC/third_party/tensorpipe/third_party/libuv/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googlemock/include -isystem /qssd/ptSRC/cmake/…/third_party/googletest/googletest/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /opt/intel/mkl/include -isystem /qssd/ptSRC/third_party/gemmlowp -isystem /qssd/ptSRC/third_party/neon2sse -isystem /qssd/ptSRC/third_party/XNNPACK/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/…/third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -O3 -DNDEBUG -fPIC -std=gnu++14 -MD -MT third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o -MF third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o.d -o third_party/tensorpipe/tensorpipe/CMakeFiles/tensorpipe.dir/channel/error.cc.o -c /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.cc
In file included from /usr/include/c++/7/ext/string_conversions.h:41:0,
from /usr/include/c++/7/bits/basic_string.h:6361,
from /usr/include/c++/7/string:52,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.h:11,
from /qssd/ptSRC/third_party/tensorpipe/tensorpipe/channel/error.cc:9:
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory
#include_next &lt;stdlib.h&gt;
^~~~~~~~~~
</code></pre>

username: studyself
cooked: <p>I am new for pytorch.<br>
Please help me.<br>
Thank you very much.</p>

username: khushi-411
cooked: <p>Hi, Could you check the output of the path of CMAKE? I think it is not configured correctly. You can use <code>echo $CMAKE_PREFIX_PATH</code> to check the path.</p>
<p>You can use this command to set up the path and then develop your environment.</p>
<pre><code class="lang-auto">export CMAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"}
</code></pre>
<p>Thanks!</p>

username: studyself
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae46280cc6b7cb87b1c63dfefaa036a98884e27c.png" data-download-href="https://discuss.pytorch.org/uploads/default/ae46280cc6b7cb87b1c63dfefaa036a98884e27c" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/e/ae46280cc6b7cb87b1c63dfefaa036a98884e27c.png" alt="image" data-base62-sha1="oRHo8EA7YxaEq2UvBQBlRrQLj8o" width="690" height="408" data-dominant-color="333434"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1227×727 67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Hi， Thank you for the reply!<br>
Is  $CMAKE_PREFIX_PATH  correct?</p>

username: khushi-411
cooked: <p>Yes, looks correct to me. I think you need to clear the cache first then rebuild. Please use: <code>python setup.py clean</code>. Then follow the build process. Also, I suggest setting flags to build your PyTorch as it takes a long time. You can use this command:</p>
<pre><code class="lang-auto">USE_DISTRIBUTED=0 USE_GLOO=0 BUILD_TEST=0 BUILD_CAFFE2=0 USE_CUDA=1 USE_CUDNN=0 USE_ASAN=0 USE_MKLDNN=0 USE_KINETO=0 DEBUG=0  MAX_JOBS=12 USE_XNNPACK=0 USE_FBGEMM=0 USE_NNPACK=0 USE_QNNPACK=0  python setup.py develop
</code></pre>
<p>Set <code>USE_FBGEMM=1</code> since you want to use that.</p>
<p>I wrote a blog ago nearly a year ago on it, here: <a href="https://khushi-411.github.io/installing-pytorch-from-source/" class="inline-onebox" rel="noopener nofollow ugc">Installing PyTorch from Source</a>. You can refer to it if you need any help!</p>

username: studyself
cooked: <p>I really clear the cache each time before building.<br>
The error is there all the way:<br>
/usr/include/c++/7/cstdlib:75:15: fatal error: stdlib.h: No such file or directory</p>

username: studyself
cooked: <p>Thank you for your message.<br>
BUT Errors hit me again</p>
<p>I think cmake configuration or gcc has problem.<br>
But I don’t know how to fix it.</p>
<pre><code class="lang-auto">[286/2044] Building CXX object third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o
FAILED: third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o
/usr/bin/c++ -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_EXTERNAL_MZCRC -D_FILE_OFFSET_BITS=64 -I/qssd/ptSRC/third_party/onnx -I/qssd/ptSRC/build/third_party/onnx -I/qssd/ptSRC/third_party/foxi -I/qssd/ptSRC/build/third_party/foxi -I/qssd/ptSRC/third_party/fmt/include -isystem /qssd/ptSRC/third_party/protobuf/src -isystem /home/quinnqiu/.conda/envs/wenetruntime/include -isystem /usr/include -isystem /qssd/ptSRC/third_party/ittapi/include -isystem /qssd/ptSRC/cmake/../third_party/eigen -I/usr/local/cuda/include -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -O3 -DNDEBUG -DNDEBUG -fPIC -DTH_HAVE_THREAD -std=gnu++14 -MD -MT third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o -MF third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o.d -o third_party/fmt/CMakeFiles/fmt.dir/src/format.cc.o -c /qssd/ptSRC/third_party/fmt/src/format.cc
In file included from /qssd/ptSRC/third_party/fmt/include/fmt/format-inl.h:14:0,
                 from /qssd/ptSRC/third_party/fmt/src/format.cc:8:
/usr/include/c++/7/cmath:45:15: fatal error: math.h: No such file or directory
 #include_next &lt;math.h&gt;
               ^~~~~~~~
compilation terminated.

</code></pre>

username: khushi-411
cooked: <p>Hmm, can you run and paste the output here?</p>
<pre><code class="lang-python">wget https://raw.githubusercontent.com/pytorch/pytorch/master/torch/utils/collect_env.py
python collect_env.py
</code></pre>
<p>Mine is: (it works fine!)</p>
<pre><code class="lang-python">Collecting environment information...
PyTorch version: 1.14.0a0+git428d647
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.5 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0
Clang version: 10.0.0-4ubuntu1 
CMake version: version 3.22.1
Libc version: glibc-2.31

Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.15.0-53-generic-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce GTX 1650 Ti
Nvidia driver version: 470.141.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: False

Versions of relevant libraries:
[pip3] mypy==0.960
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.21.6
[pip3] torch==1.14.0a0+gitc1cf006
[conda] mkl                       2022.1.0           hc2b9512_224  
[conda] mkl-include               2022.1.0           h06a4308_224  
[conda] numpy                     1.21.6                   pypi_0    pypi
[conda] torch                     1.14.0a0+git0efd4e9          pypi_0    pypi
</code></pre>

username: studyself
cooked: <p>Should I update my enviroments?</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.22.1
Libc version: glibc-2.27

Python version: 3.9.15 (main, Nov  4 2022, 16:13:54)  [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.27
Is CUDA available: N/A
CUDA runtime version: 11.2.67
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration:
GPU 0: GeForce GTX 1070
GPU 1: GeForce GTX 1070

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A

Versions of relevant libraries:
[pip3] numpy==1.23.4
[conda] blas                      1.0                         mkl
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2021.4.0           h06a4308_640
[conda] mkl-include               2022.1.0           h06a4308_224
[conda] mkl-service               2.4.0            py39h7f8727e_0
[conda] mkl_fft                   1.3.1            py39hd3c417c_0
[conda] mkl_random                1.2.2            py39h51133e4_0
[conda] numpy                     1.23.4           py39h14f4228_0
[conda] numpy-base                1.23.4           py39h31eccc5_0

</code></pre>

username: studyself
cooked: <p>I have successfully built pytorch from source code on the same machine, but with a different account.<br>
It is strange that I login with the other account, Errors occur.<br>
I try to use the same version of cmake, but it didn’t help.<br>
I try to use the same version of python, neither help.</p>
<p>Just FYI. The successful python collect_env.py:</p>
<pre><code class="lang-auto">Collecting environment information...
PyTorch version: 1.14.0a0+gita80e5e7
Is debug build: False
CUDA used to build PyTorch: 11.2
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: 6.0.0-1ubuntu2 (tags/RELEASE_600/final)
CMake version: version 3.25.0
Libc version: glibc-2.27

Python version: 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:18)  [GCC 10.3.0] (64-bit runtime)
Python platform: Linux-4.15.0-91-generic-x86_64-with-glibc2.10
Is CUDA available: True
CUDA runtime version: 11.2.67
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: GeForce GTX 1070
GPU 1: GeForce GTX 1070

Nvidia driver version: 460.91.03
cuDNN version: Probably one of the following:
/usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudnn.so.7.6.0
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.7
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_adv_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_cnn_train.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_infer.so.8
/usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudnn_ops_train.so.8
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

Versions of relevant libraries:
[pip3] numpy==1.23.5
[pip3] torch==1.14.0a0+gita80e5e7
[conda] magma-cuda112             2.5.2                         1    pytorch
[conda] mkl                       2022.1.0           hc2b9512_224    http://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main
[conda] mkl-include               2022.1.0           h84fe81f_915    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] numpy                     1.23.5           py38h7042d01_0    http://mirrors.tuna.tsinghua.edu.cn/anaconda/cloud/conda-forge
[conda] torch                     1.14.0a0+gita80e5e7           dev_0    &lt;develop&gt;

</code></pre>