Build torch from source failed

username: urningod
cooked: <p>Hi,</p>
<p>I am trying to build torch from source in a docker. But the building process failed.</p>
<p>I followed <a href="https://github.com/pytorch/pytorch#from-source" rel="noopener nofollow ugc">this document</a> to build torch (CPU), and I have ran the following commands (I didn’t use conda because I am building in a docker):</p>
<pre><code class="lang-auto">pip install astunparse numpy ninja pyyaml setuptools cmake cffi typing_extensions future six requests dataclasses
pip install mkl mkl-include
git clone --recursive https://github.com/pytorch/pytorch
cd pytorch
git checkout xxxx
git submodule sync
git submodule update --init --recursive --jobs 0
python setup.py install
</code></pre>
<p>Then I got this error when running <code>python setup.py install</code></p>
<pre><code class="lang-auto">[5217/6244] Building C object caffe2/CMakeFiles/torch_cpu.dir/__/third_party/miniz-2.1.0/miniz.c.o
FAILED: caffe2/CMakeFiles/torch_cpu.dir/__/third_party/miniz-2.1.0/miniz.c.o 
/usr/bin/cc -DAT_PER_OPERATOR_HEADERS -DBUILD_ONEDNN_GRAPH -DCPUINFO_SUPPORTED_PLATFORM=1 -DFMT_HEADER_ONLY=1 -DFXDIV_USE_INLINE_ASSEMBLY=0 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MMAP=1 -DHAVE_SHM_OPEN=1 -DHAVE_SHM_UNLINK=1 -DMINIZ_DISABLE_ZIP_READER_CRC32_CHECKS -DNNP_CONVOLUTION_ONLY=0 -DNNP_INFERENCE_ONLY=0 -DONNXIFI_ENABLE_EXT=1 -DONNX_ML=1 -DONNX_NAMESPACE=onnx_torch -DUSE_C10D_GLOO -DUSE_DISTRIBUTED -DUSE_EXTERNAL_MZCRC -DUSE_RPC -DUSE_TENSORPIPE -D_FILE_OFFSET_BITS=64 -Dtorch_cpu_EXPORTS -I/home/code/pytorch/build/aten/src -I/home/code/pytorch/aten/src -I/home/code/pytorch/build -I/home/code/pytorch -I/home/code/pytorch/cmake/../third_party/benchmark/include -I/home/code/pytorch/third_party/onnx -I/home/code/pytorch/build/third_party/onnx -I/home/code/pytorch/third_party/foxi -I/home/code/pytorch/build/third_party/foxi -I/home/code/pytorch/torch/csrc/api -I/home/code/pytorch/torch/csrc/api/include -I/home/code/pytorch/caffe2/aten/src/TH -I/home/code/pytorch/build/caffe2/aten/src/TH -I/home/code/pytorch/build/caffe2/aten/src -I/home/code/pytorch/build/caffe2/../aten/src -I/home/code/pytorch/torch/csrc -I/home/code/pytorch/third_party/miniz-2.1.0 -I/home/code/pytorch/third_party/kineto/libkineto/include -I/home/code/pytorch/third_party/kineto/libkineto/src -I/home/code/pytorch/torch/csrc/distributed -I/home/code/pytorch/aten/../third_party/catch/single_include -I/home/code/pytorch/aten/src/ATen/.. -I/home/code/pytorch/third_party/FXdiv/include -I/home/code/pytorch/c10/.. -I/home/code/pytorch/third_party/pthreadpool/include -I/home/code/pytorch/third_party/cpuinfo/include -I/home/code/pytorch/third_party/QNNPACK/include -I/home/code/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/include -I/home/code/pytorch/aten/src/ATen/native/quantized/cpu/qnnpack/src -I/home/code/pytorch/third_party/cpuinfo/deps/clog/include -I/home/code/pytorch/third_party/NNPACK/include -I/home/code/pytorch/third_party/fbgemm/include -I/home/code/pytorch/third_party/fbgemm -I/home/code/pytorch/third_party/fbgemm/third_party/asmjit/src -I/home/code/pytorch/third_party/FP16/include -I/home/code/pytorch/third_party/tensorpipe -I/home/code/pytorch/build/third_party/tensorpipe -I/home/code/pytorch/third_party/tensorpipe/third_party/libnop/include -I/home/code/pytorch/third_party/fmt/include -I/home/code/pytorch/build/third_party/ideep/mkl-dnn/third_party/oneDNN/include -I/home/code/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN/src/../include -I/home/code/pytorch/third_party/flatbuffers/include -isystem /home/code/pytorch/build/third_party/gloo -isystem /home/code/pytorch/cmake/../third_party/gloo -isystem /home/code/pytorch/cmake/../third_party/googletest/googlemock/include -isystem /home/code/pytorch/cmake/../third_party/googletest/googletest/include -isystem /home/code/pytorch/third_party/protobuf/src -isystem /home/code/pytorch/third_party/gemmlowp -isystem /home/code/pytorch/third_party/neon2sse -isystem /home/code/pytorch/third_party/XNNPACK/include -isystem /home/code/pytorch/cmake/../third_party/eigen -isystem /home/code/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN/include -isystem /home/code/pytorch/third_party/ideep/include -isystem /home/code/pytorch/third_party/ideep/mkl-dnn/include -isystem /home/code/pytorch/build/include -fopenmp -DNDEBUG -O3 -DNDEBUG -DNDEBUG -fPIC -DCAFFE2_USE_GLOO -DTH_HAVE_THREAD -fopenmp=libomp -DCAFFE2_BUILD_MAIN_LIB -pthread -DASMJIT_STATIC -std=gnu11 -MD -MT caffe2/CMakeFiles/torch_cpu.dir/__/third_party/miniz-2.1.0/miniz.c.o -MF caffe2/CMakeFiles/torch_cpu.dir/__/third_party/miniz-2.1.0/miniz.c.o.d -o caffe2/CMakeFiles/torch_cpu.dir/__/third_party/miniz-2.1.0/miniz.c.o -c /home/code/pytorch/third_party/miniz-2.1.0/miniz.c
cc: error: unrecognized command line option ‘-fopenmp=libomp’; did you mean ‘-fopenmp-simd’?
[5274/6244] Building CXX object caffe2/CMakeFiles/torch_cpu.dir/__/aten/src/ATen/RegisterCPU.cpp.o
ninja: build stopped: subcommand failed.

</code></pre>
<p>Here is the summary when running  <code>python setup.py install</code>:</p>
<pre><code class="lang-auto">-- ******** Summary ********
-- General:
--   CMake version         : 3.24.1
--   CMake command         : /root/miniconda3/lib/python3.7/site-packages/cmake/data/bin/cmake
--   System                : Linux
--   C++ compiler          : /usr/bin/c++
--   C++ compiler id       : Clang
--   C++ compiler version  : 6.0.0
--   Using ccache if found : ON
--   Found ccache          : CCACHE_PROGRAM-NOTFOUND
--   CXX flags             :  -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp=libomp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-range-loop-analysis -Wno-pass-failed -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -Wconstant-conversion -Wno-invalid-partial-specialization -Wno-typedef-redefinition -Wno-unknown-warning-option -Wno-unused-private-field -Wno-inconsistent-missing-override -Wno-aligned-allocation-unavailable -Wno-c++14-extensions -Wno-constexpr-not-const -Wno-missing-braces -Qunused-arguments -fcolor-diagnostics -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type
--   Build type            : Release
--   Compile definitions   : ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : /root/miniconda3/lib/python3.7/site-packages
--   CMAKE_INSTALL_PREFIX  : /home/code/pytorch/torch
--   USE_GOLD_LINKER       : OFF
-- 
--   TORCH_VERSION         : 1.13.0
--   CAFFE2_VERSION        : 1.13.0
--   BUILD_CAFFE2          : OFF
--   BUILD_CAFFE2_OPS      : OFF
--   BUILD_CAFFE2_MOBILE   : OFF
--   BUILD_STATIC_RUNTIME_BENCHMARK: OFF
--   BUILD_TENSOREXPR_BENCHMARK: OFF
--   BUILD_NVFUSER_BENCHMARK: ON
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.7.6
--     Python executable   : /root/miniconda3/bin/python
--     Pythonlibs version  : 3.7.6
--     Python library      : /root/miniconda3/lib/libpython3.7m.so.1.0
--     Python includes     : /root/miniconda3/include/python3.7m
--     Python site-packages: lib/python3.7/site-packages
--   BUILD_SHARED_LIBS     : ON
--   CAFFE2_USE_MSVC_STATIC_RUNTIME     : OFF
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   BUILD_MOBILE_AUTOGRAD : OFF
--   BUILD_LITE_INTERPRETER: OFF
--   INTERN_BUILD_MOBILE   : 
--   USE_BLAS              : 0
--   USE_LAPACK            : 0
--   USE_ASAN              : OFF
--   USE_CPP_CODE_COVERAGE : OFF
--   USE_CUDA              : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : ON
--   USE_FBGEMM            : ON
--     USE_FAKELOWP          : OFF
--   USE_KINETO            : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_PYTORCH_METAL     : OFF
--   USE_PYTORCH_METAL_EXPORT     : OFF
--   USE_MPS               : OFF
--   USE_FFTW              : OFF
--   USE_MKL               : OFF
--   USE_MKLDNN            : ON
--   USE_NCCL              : OFF
--   USE_NNPACK            : ON
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_VULKAN            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_PYTORCH_QNNPACK   : ON
--   USE_XNNPACK           : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : ON
--     USE_MPI               : OFF
--     USE_GLOO              : ON
--     USE_GLOO_WITH_OPENSSL : OFF
--     USE_TENSORPIPE        : ON
--   USE_DEPLOY           : OFF
--   Public Dependencies  : caffe2::Threads
--   Private Dependencies : pthreadpool;cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;fp16;tensorpipe;gloo;foxi_loader;rt;fmt::fmt-header-only;kineto;dl
--   USE_COREML_DELEGATE     : OFF
--   BUILD_LAZY_TS_BACKEND   : ON
-- Configuring done
-- Generating done
-- Build files have been written to: /home/code/pytorch/build
</code></pre>
<p>Environment:<br>
OS: Ubuntu 18.04.4 LTS (x86_64)</p>
<p>I wonder what did I miss here? Thank you!</p>

username: ptrblck
cooked: <p>Could you check the build log for previous error messages and post them here, please?</p>

username: urningod
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thanks for looking into this!</p>
<p>The full log when running <code>python setup.py install</code> can be found in this <a href="https://gist.github.com/DNXie/0d5e5d0d922f27ca208477f34fb39b5b" rel="noopener nofollow ugc">gist</a>.</p>
<p>Here is the Dockerfile I use to build the docker container:</p>
<pre><code class="lang-auto">ARG UBUNTU_VERSION=18.04
FROM ubuntu:${UBUNTU_VERSION}

# Install required ubuntu packages
RUN apt-get update &amp;&amp; apt-get install -y screen vim wget git &amp;&amp; rm -rf /var/lib/apt/lists/*
RUN apt-get update &amp;&amp; apt-get -y install gcc clang clang-tools cmake protobuf-compiler

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    &amp;&amp; mkdir /root/.conda \
    &amp;&amp; bash Miniconda3-latest-Linux-x86_64.sh -b \
    &amp;&amp; rm -f Miniconda3-latest-Linux-x86_64.sh 
ENV PATH /root/miniconda3/bin:$PATH

RUN touch /root/.vimrc
RUN echo 'syntax on\nset tabstop=2\nset softtabstop=2\nset showcmd\nset showmatch\nset incsearch\nset hlsearch\nset ruler'&gt;&gt; /root/.vimrc


RUN conda install      \
          pip=20.0.2   \
          python=3.7.6 \
          scikit-learn=0.22.1 \
          pandas=1.3.5
        
CMD /bin/bash
</code></pre>
<p>Thank you so much!</p>

username: ptrblck
cooked: <p>The error:</p>
<pre><code class="lang-auto">cc: error: unrecognized command line option ‘-fopenmp=libomp’; did you mean ‘-fopenmp-simd’?
</code></pre>
<p>points to e.g. <a href="https://stackoverflow.com/questions/67689069/linking-cmake-target-with-openmp-in-a-mixed-c-c-project-with-different-compile">this issue</a> which can be caused by mixing compilers.</p>
<p>I’m not familiar with your setup, but are you mixing <code>clang</code> with <code>gcc</code>?</p>
<pre><code class="lang-auto">Detected C compiler: GNU @ /usr/bin/cc
...
--   C++ compiler          : /usr/bin/c++
--   C++ compiler id       : Clang
...
</code></pre>

username: urningod
cooked: <p>Yes it seems so. Thanks for pointing it out.<br>
I tried <code>ln -s /usr/bin/c++ /usr/bin/cc</code> and it solves the problem!<br>
Really appreciate your help!</p>