Compile PyTorch with MPI as conda package

css: <p><strong>What I am trying</strong><br>
I am trying to compile PyTorch with MPI, CUDA and CUDNN support and package it as a conda package.<br>
As a first step I just want to compile against an arbitrary IntelMPI and CUDA/CUDNN version and get a working package.<br>
Unfortunately there are not many good guides out there that explain the process.</p>
<p><strong>My questions</strong></p>
<ul>
<li>Do you know any good guide which explains how to compile pytorch with mpi, cuda and cudnn support and package it with conda?</li>
<li>Where are the recipes used by the pytorchbot, which uploads the pytorch packages to anaconda?</li>
<li>How do I fix these <code>WARNING (pytorch,lib/python3.8/site-packages/torch/test/cpuid_test): $RPATH/libtorch_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist. </code> errors?</li>
<li>Why does the package process abort? I have no clue why it does not succeed.</li>
<li>Is it normal that the compilation of CUDA NVCC uses just one core?</li>
</ul>
<p><strong>Status quo</strong><br>
The compilation ends successfully, the packaging not.<br>
IntelMPI, CUDNN and CUDA are all found correctly in the configure process (see install.log)<br>
The <code>meta.yaml</code> looks like this (I know that I dont need all of these packages at runtime, but I just want to make it work as a first step):</p>
<pre><code class="lang-auto">package:
  name: pytorch
  version: 1.10.0

source:
  git_rev: v1.10.0
  git_url: https://github.com/pytorch/pytorch.git

requirements:
  build:
    - python
    - setuptools
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

  run:
    - python
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

  host:
    - python
    - astunparse
    - numpy
    - ninja
    - pyyaml
    - mkl
    - mkl-include
    - setuptools
    - cmake
    - cffi
    - typing_extensions
    - future
    - six
    - requests
    - dataclasses

</code></pre>
<p>The <code>build.sh</code> looks like this:</p>
<pre><code class="lang-auto">#!/usr/bin/env bash

set -x
set -e
set -u

$PYTHON setup.py clean 2&gt;&amp;1
USE_NINJA=0 USE_CUDA=1 USE_CUDNN=1 USE_NUMPY=1 USE_DISTRIBUTED=1 USE_MPI=1 USE_GLOO=1 USE_SYSTEM_NCCL=0 MAX_JOBS=$(nproc) MAKE_PREFIX_PATH=${CONDA_PREFIX:-"$(dirname $(which conda))/../"} $PYTHON -u setup.py install --cmake 2&gt;&amp;1
echo "Successfully biuld pytorch"
</code></pre>
<p>and I run conda-build like this</p>
<pre><code class="lang-auto">conda-build . --keep-old-work --dirty |&amp; tee install.log
</code></pre>
<p>The important parts of the  <code>install.log</code> are:</p>
<pre><code class="lang-auto">## Package Plan ##

  environment location: /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_build_env


The following NEW packages will be INSTALLED:

    _libgcc_mutex:      0.1-main                
    _openmp_mutex:      4.5-1_gnu               
    astunparse:         1.6.3-py_0              
    blas:               1.0-mkl                 
    brotlipy:           0.7.0-py38h27cfd23_1003 
    bzip2:              1.0.8-h7b6447c_0        
    c-ares:             1.17.1-h27cfd23_0       
    ca-certificates:    2021.10.26-h06a4308_2   
    certifi:            2021.10.8-py38h06a4308_0
    cffi:               1.14.6-py38h400218f_0   
    charset-normalizer: 2.0.4-pyhd3eb1b0_0      
    cmake:              3.19.6-h973ab73_0       
    cryptography:       35.0.0-py38hd23ed53_0   
    dataclasses:        0.8-pyh6d0b6a4_7        
    expat:              2.4.1-h2531618_2        
    future:             0.18.2-py38_1           
    idna:               3.2-pyhd3eb1b0_0        
    intel-openmp:       2021.4.0-h06a4308_3561  
    krb5:               1.19.2-hac12032_0       
    ld_impl_linux-64:   2.35.1-h7274673_9       
    libcurl:            7.78.0-h0b77cf5_0       
    libedit:            3.1.20210910-h7f8727e_0 
    libev:              4.33-h7f8727e_1         
    libffi:             3.3-he6710b0_2          
    libgcc-ng:          9.3.0-h5101ec6_17       
    libgfortran-ng:     7.5.0-ha8ba4b0_17       
    libgfortran4:       7.5.0-ha8ba4b0_17       
    libgomp:            9.3.0-h5101ec6_17       
    libnghttp2:         1.41.0-hf8bcb03_2       
    libssh2:            1.9.0-h1ba5d50_1        
    libstdcxx-ng:       9.3.0-hd4cf53a_17       
    libuv:              1.40.0-h7b6447c_0       
    lz4-c:              1.9.3-h295c915_1        
    mkl:                2021.4.0-h06a4308_640   
    mkl-include:        2021.4.0-h06a4308_640   
    mkl-service:        2.4.0-py38h7f8727e_0    
    mkl_fft:            1.3.1-py38hd3c417c_0    
    mkl_random:         1.2.2-py38h51133e4_0    
    ncurses:            6.3-h7f8727e_2          
    ninja:              1.10.2-py38hd09550d_3   
    numpy:              1.16.6-py38h2d18471_3   
    numpy-base:         1.16.6-py38hdc34a94_3   
    openssl:            1.1.1l-h7f8727e_0       
    pip:                21.2.4-py38h06a4308_0   
    pycparser:          2.21-pyhd3eb1b0_0       
    pyopenssl:          21.0.0-pyhd3eb1b0_1     
    pysocks:            1.7.1-py38h06a4308_0    
    python:             3.8.12-h12debd9_0       
    pyyaml:             6.0-py38h7f8727e_1      
    readline:           8.1-h27cfd23_0          
    requests:           2.26.0-pyhd3eb1b0_0     
    rhash:              1.4.1-h3c74f83_1        
    setuptools:         58.0.4-py38h06a4308_0   
    six:                1.16.0-pyhd3eb1b0_0     
    sqlite:             3.36.0-hc218d9a_0       
    tk:                 8.6.11-h1ccaba5_0       
    typing_extensions:  3.10.0.2-pyh06a4308_0   
    urllib3:            1.26.7-pyhd3eb1b0_0     
    wheel:              0.37.0-pyhd3eb1b0_1     
    xz:                 5.2.5-h7b6447c_0        
    yaml:               0.2.5-h7b6447c_0        
    zlib:               1.2.11-h7b6447c_3       
    zstd:               1.4.9-haebb681_0        

Preparing transaction: ...working... done
Verifying transaction: ...working... done
Executing transaction: ...working... done
+ set -e
+ set -u
+ /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/python setup.py clean
+ USE_NINJA=0
+ USE_CUDA=1
+ USE_CUDNN=1
+ USE_NUMPY=1
+ USE_DISTRIBUTED=1
+ USE_MPI=1
+ USE_GLOO=1
+ USE_SYSTEM_NCCL=0
++ nproc
+ MAX_JOBS=64
+ MAKE_PREFIX_PATH=/anaconda/envs/compile/conda-bld/pytorch_1636708344734/_build_env
+ /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/bin/python -u setup.py install --cmake
source tree in: /anaconda/envs/compile/conda-bld/pytorch_1636708344734/work
export PREFIX=/anaconda/envs/compile/conda-bld/pytorch_1636708344734
</code></pre>
<pre><code class="lang-auto">--   Build type            : Release
--   Compile definitions   : TH_BLAS_MKL;ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;IDEEP_USE_MKL;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS
--   CMAKE_PREFIX_PATH     : $PREFIX/lib/python3.8/site-packages;/usr/local/cuda
--   CMAKE_INSTALL_PREFIX  : $SRC_DIR/torch
--   USE_GOLD_LINKER       : OFF
-- 
--   TORCH_VERSION         : 1.10.0
--   CAFFE2_VERSION        : 1.10.0
--   BUILD_CAFFE2          : ON
--   BUILD_CAFFE2_OPS      : ON
--   BUILD_CAFFE2_MOBILE   : OFF
--   BUILD_STATIC_RUNTIME_BENCHMARK: OFF
--   BUILD_TENSOREXPR_BENCHMARK: OFF
--   BUILD_BINARY          : OFF
--   BUILD_CUSTOM_PROTOBUF : ON
--     Link local protobuf : ON
--   BUILD_DOCS            : OFF
--   BUILD_PYTHON          : True
--     Python version      : 3.8.12
--     Python executable   : $PREFIX/bin/python
--     Pythonlibs version  : 3.8.12
--     Python library      : $PREFIX/lib/libpython3.8.so.1.0
--     Python includes     : $PREFIX/include/python3.8
--     Python site-packages: lib/python3.8/site-packages
--   BUILD_SHARED_LIBS     : ON
--   CAFFE2_USE_MSVC_STATIC_RUNTIME     : OFF
--   BUILD_TEST            : True
--   BUILD_JNI             : OFF
--   BUILD_MOBILE_AUTOGRAD : OFF
--   BUILD_LITE_INTERPRETER: OFF
--   INTERN_BUILD_MOBILE   : 
--   USE_BLAS              : 1
--     BLAS                : mkl
--   USE_LAPACK            : 1
--     LAPACK              : mkl
--   USE_ASAN              : OFF
--   USE_CPP_CODE_COVERAGE : OFF
--   USE_CUDA              : 1
--     Split CUDA          : OFF
--     CUDA static link    : OFF
--     USE_CUDNN           : 1
--     USE_EXPERIMENTAL_CUDNN_V8_API: OFF
--     CUDA version        : 11.1
--     cuDNN version       : 8.1.1
--     CUDA root directory : /usr/local/cuda
--     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so
--     cudart library      : /usr/local/cuda/lib64/libcudart.so
--     cublas library      : /usr/local/cuda/lib64/libcublas.so
--     cufft library       : /usr/local/cuda/lib64/libcufft.so
--     curand library      : /usr/local/cuda/lib64/libcurand.so
--     cuDNN library       : /usr/lib/x86_64-linux-gnu/libcudnn.so
--     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so
--     CUDA include path   : /usr/local/cuda/include
--     NVCC executable     : /usr/local/cuda/bin/nvcc
--     NVCC flags          : -Xfatbin;-compress-all;-DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_52,code=sm_52;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_86,code=compute_86;-Xcudafe;--diag_suppress=cc_clobber_ignored,--diag_suppress=integer_sign_change,--diag_suppress=useless_using_declaration,--diag_suppress=set_but_not_used,--diag_suppress=field_without_dll_interface,--diag_suppress=base_class_has_different_dll_interface,--diag_suppress=dll_interface_conflict_none_assumed,--diag_suppress=dll_interface_conflict_dllexport_assumed,--diag_suppress=implicit_return_from_non_void_function,--diag_suppress=unsigned_compare_with_zero,--diag_suppress=declared_but_not_referenced,--diag_suppress=bad_friend_decl;-std=c++14;-Xcompiler;-fPIC;--expt-relaxed-constexpr;--expt-extended-lambda;-Wno-deprecated-gpu-targets;--expt-extended-lambda;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_BFLOAT16_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__
--     CUDA host compiler  : /usr/bin/cc
--     NVCC --device-c     : OFF
--     USE_TENSORRT        : OFF
--   USE_ROCM              : OFF
--   USE_EIGEN_FOR_BLAS    : 
--   USE_FBGEMM            : ON
--     USE_FAKELOWP          : OFF
--   USE_KINETO            : ON
--   USE_FFMPEG            : OFF
--   USE_GFLAGS            : OFF
--   USE_GLOG              : OFF
--   USE_LEVELDB           : OFF
--   USE_LITE_PROTO        : OFF
--   USE_LMDB              : OFF
--   USE_METAL             : OFF
--   USE_PYTORCH_METAL     : OFF
--   USE_PYTORCH_METAL_EXPORT     : OFF
--   USE_FFTW              : OFF
--   USE_MKL               : ON
--   USE_MKLDNN            : ON
--   USE_MKLDNN_ACL        : OFF
--   USE_MKLDNN_CBLAS      : OFF
--   USE_NCCL              : ON
--     USE_SYSTEM_NCCL     : 0
--   USE_NNPACK            : ON
--   USE_NUMPY             : ON
--   USE_OBSERVERS         : ON
--   USE_OPENCL            : OFF
--   USE_OPENCV            : OFF
--   USE_OPENMP            : ON
--   USE_TBB               : OFF
--   USE_VULKAN            : OFF
--   USE_PROF              : OFF
--   USE_QNNPACK           : ON
--   USE_PYTORCH_QNNPACK   : ON
--   USE_REDIS             : OFF
--   USE_ROCKSDB           : OFF
--   USE_ZMQ               : OFF
--   USE_DISTRIBUTED       : 1
--     USE_MPI               : 1
--     USE_GLOO              : 1
--     USE_GLOO_WITH_OPENSSL : OFF
--     USE_TENSORPIPE        : ON
--   USE_DEPLOY           : OFF
--   USE_BREAKPAD         : ON
--   Public Dependencies  : Threads::Threads;caffe2::mkl;caffe2::mkldnn
--   Private Dependencies : pthreadpool;cpuinfo;qnnpack;pytorch_qnnpack;nnpack;XNNPACK;fbgemm;/usr/lib/x86_64-linux-gnu/libnuma.so;fp16;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/libmpicxx.so;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/release_mt/libmpi.so;/opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/libmpigi.a;/usr/lib/x86_64-linux-gnu/libdl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/x86_64-linux-gnu/libpthread.so;gloo;tensorpipe;aten_op_header_gen;foxi_loader;rt;fmt::fmt-header-only;kineto;gcc_s;gcc;dl
--   USE_COREML_DELEGATE     : OFF
-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    BUILD_PREFIX
    USE_NINJA


-- Build files have been written to: $SRC_DIR/build
cmake --build . --target install --config Release -- -j 64
Scanning dependencies of target nccl_external
Scanning dependencies of target defs.bzl
Scanning dependencies of target clog
Scanning dependencies of target libkineto_defs.bzl
Scanning dependencies of target ATEN_CPU_FILES_GEN_TARGET
Scanning dependencies of target nvfuser_rt_fp16_support
Scanning dependencies of target ATEN_CUDA_FILES_GEN_TARGET
Scanning dependencies of target mkdisp
Scanning dependencies of target nvfuser_rt_tensor
Scanning dependencies of target torch_global_deps
Scanning dependencies of target nvfuser_rt_block_reduction
Scanning dependencies of target nvfuser_rt_random_numbers
Scanning dependencies of target foxi_loader
Scanning dependencies of target mkrename
Scanning dependencies of target pthreadpool
Scanning dependencies of target nvfuser_rt_helpers
Scanning dependencies of target nvfuser_rt_grid_reduction
Scanning dependencies of target mkalias
Scanning dependencies of target nvfuser_rt_UnpackRaw
Scanning dependencies of target nvfuser_rt_PhiloxCudaStateRaw
Scanning dependencies of target gen_torch_version
Scanning dependencies of target nvfuser_rt_broadcast
[  0%] Creating directories for 'nccl_external'
Scanning dependencies of target gtest
Scanning dependencies of target common
Scanning dependencies of target mkrename_gnuabi
Scanning dependencies of target mkmasked_gnuabi
Scanning dependencies of target arraymap
[  0%] Built target defs.bzl
</code></pre>
<pre><code class="lang-auto">[  0%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CPUFunctions_inl.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions_inl.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions_inl.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/MetaFunctions_inl.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/NativeMetaFunctions.h, ../aten/src/ATen/Operators.h, ../aten/src/ATen/Operators_0.cpp, ../aten/src/ATen/Operators_1.cpp, ../aten/src/ATen/Operators_2.cpp, ../aten/src/ATen/Operators_3.cpp, ../aten/src/ATen/Operators_4.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/CUDAFunctions_inl.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
Scanning dependencies of target fmt
[  0%] Building C object confu-deps/cpuinfo/deps/clog/CMakeFiles/clog.dir/src/clog.c.o
Scanning dependencies of target generate-torch-sources
Scanning dependencies of target kineto_api
[  0%] Built target libkineto_defs.bzl
Scanning dependencies of target python_copy_files
Scanning dependencies of target addSuffix
[  0%] Building C object sleef/src/libm/CMakeFiles/mkrename.dir/mkrename.c.o
[  0%] Building C object sleef/src/libm/CMakeFiles/mkdisp.dir/mkdisp.c.o
[  0%] Building C object third_party/foxi/CMakeFiles/foxi_loader.dir/foxi/onnxifi_loader.c.o
[  0%] Building C object caffe2/CMakeFiles/torch_global_deps.dir/__/torch/csrc/empty.c.o
[  0%] Generating ../aten/src/ATen/CPUFunctions.h, ../aten/src/ATen/CPUFunctions_inl.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions.h, ../aten/src/ATen/CompositeExplicitAutogradFunctions_inl.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions.h, ../aten/src/ATen/CompositeImplicitAutogradFunctions_inl.h, ../aten/src/ATen/Declarations.yaml, ../aten/src/ATen/Functions.cpp, ../aten/src/ATen/Functions.h, ../aten/src/ATen/MetaFunctions.h, ../aten/src/ATen/MetaFunctions_inl.h, ../aten/src/ATen/NativeFunctions.h, ../aten/src/ATen/NativeMetaFunctions.h, ../aten/src/ATen/Operators.h, ../aten/src/ATen/Operators_0.cpp, ../aten/src/ATen/Operators_1.cpp, ../aten/src/ATen/Operators_2.cpp, ../aten/src/ATen/Operators_3.cpp, ../aten/src/ATen/Operators_4.cpp, ../aten/src/ATen/RedispatchFunctions.h, ../aten/src/ATen/RegisterBackendSelect.cpp, ../aten/src/ATen/RegisterCPU.cpp, ../aten/src/ATen/RegisterCompositeExplicitAutograd.cpp, ../aten/src/ATen/RegisterCompositeImplicitAutograd.cpp, ../aten/src/ATen/RegisterMeta.cpp, ../aten/src/ATen/RegisterMkldnnCPU.cpp, ../aten/src/ATen/RegisterQuantizedCPU.cpp, ../aten/src/ATen/RegisterSchema.cpp, ../aten/src/ATen/RegisterSparseCPU.cpp, ../aten/src/ATen/RegisterSparseCsrCPU.cpp, ../aten/src/ATen/RegistrationDeclarations.h, ../aten/src/ATen/CUDAFunctions.h, ../aten/src/ATen/CUDAFunctions_inl.h, ../aten/src/ATen/RegisterCUDA.cpp, ../aten/src/ATen/RegisterQuantizedCUDA.cpp, ../aten/src/ATen/RegisterSparseCUDA.cpp, ../aten/src/ATen/RegisterSparseCsrCUDA.cpp, ../aten/src/ATen/core/ATenOpList.cpp, ../aten/src/ATen/core/TensorBody.h, ../aten/src/ATen/core/TensorMethods.cpp
[  0%] Building CXX object third_party/googletest/googletest/CMakeFiles/gtest.dir/src/gtest-all.cc.o
[100%] Built target caffe2_detectron_ops_gpu
Install the project...
-- Install configuration: "Release"
-- Set runtime path of "$SRC_DIR/torch/bin/protoc-3.13.0.0" to "$ORIGIN/../lib"
-- Set runtime path of "$SRC_DIR/torch/lib/libc10.so" to "$ORIGIN"
-- Set runtime path of "$SRC_DIR/torch/test/c10_CompileTimeFunctionPointer_test" to "$ORIGIN"
creating build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/throughput_benchmark.py -&gt; build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/mkldnn.py -&gt; build/lib.linux-x86_64-3.8/torch/utils
copying torch/utils/mobile_optimizer.py -&gt; build/lib.linux-x86_64-3.8/torch/utils
</code></pre>
<pre><code class="lang-auto">/intel64', '/usr/local/cuda/lib64'] disagree for /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla/lib/python3.8/site-packages/torch/test/pow_test :: 
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/compilers_and_libraries_2018.3.222/linux/mpi/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib/release_mt is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
Warning: rpath /opt/intel/mpi-rt/2017.0.0/intel64/lib is outside prefix /anaconda/envs/compile/conda-bld/pytorch_1636708344734/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_pla (removing it)
</code></pre>
<pre><code class="lang-auto">
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): Needed DSO lib/libstdc++.so.6 found in ['libstdcxx-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): .. but ['libstdcxx-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): Needed DSO lib/libgcc_s.so.1 found in ['libgcc-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): .. but ['libgcc-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/net_async_tracing_test): $RPATH/libc.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libcudart.so.11.0 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch_cpu.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libtorch_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc10_cuda.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc10.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): Needed DSO lib/libstdc++.so.6 found in ['libstdcxx-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): .. but ['libstdcxx-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libm.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): Needed DSO lib/libgcc_s.so.1 found in ['libgcc-ng']
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): .. but ['libgcc-ng'] not in reqs/run, (i.e. it is overlinking) (likely) or a missing dependency (less likely)
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libpthread.so.0 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/cuda_atomic_ops_test): $RPATH/libc.so.6 not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/graph_test): $RPATH/libtorch.so not found in packages, sysroot(s) nor the missing_dso_whitelist.
.. is this binary repackaging?
WARNING (pytorch,lib/python3.8/site-packages/torch/test/graph_test): $RPATH/libtorch_cpu.so not 
</code></pre>
<pre><code class="lang-auto">[  1%] Running gen_proto.py on onnx/onnx.in.proto
[  3%] Building C object CMakeFiles/onnxifi_dummy.dir/onnx/onnxifi_dummy.c.o
[  5%] Building C object CMakeFiles/onnxifi_loader.dir/onnx/onnxifi_loader.c.o
Processing $SRC_DIR/onnx/onnx.in.proto
Writing $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto
Writing $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto3
generating $SRC_DIR/.setuptools-cmake-build/onnx/onnx_pb.py
[  7%] Running C++ protocol buffer compiler on $SRC_DIR/.setuptools-cmake-build/onnx/onnx.proto
[  9%] Linking C static library libonnxifi_loader.a
[ 11%] Linking C shared library libonnxifi_dummy.so
[ 11%] Built target onnxifi_loader
Scanning dependencies of target onnxifi_wrapper
[ 12%] Building C object CMakeFiles/onnxifi_wrapper.dir/onnx/onnxifi_wrapper.c.o
[ 12%] Built target onnxifi_dummy
CMakeFiles/gen_onnx_proto.dir/build.make:61: recipe for target 'onnx/onnx.pb.cc' failed
CMakeFiles/Makefile2:205: recipe for target 'CMakeFiles/gen_onnx_proto.dir/all' failed
[ 14%] Linking C shared module libonnxifi.so
[ 14%] Built target onnxifi_wrapper
Makefile:129: recipe for target 'all' failed
Traceback (most recent call last):
  File "/anaconda/envs/compile/bin/conda-build", line 11, in &lt;module&gt;
    sys.exit(main())
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 480, in main
    execute(sys.argv[1:])
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/cli/main_build.py", line 469, in execute
    outputs = api.build(args.recipe, post=args.post, test_run_post=args.test_run_post,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/api.py", line 186, in build
    return build_tree(
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/build.py", line 3083, in build_tree
    packages_from_this = build(metadata, stats,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/build.py", line 2206, in build
    utils.check_call_env(cmd, env=env, rewrite_stdout_env=rewrite_env,
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/utils.py", line 410, in check_call_env
    return _func_defaulting_env_to_os_environ('call', *popenargs, **kwargs)
  File "/anaconda/envs/compile/lib/python3.8/site-packages/conda_build/utils.py", line 390, in _func_defaulting_env_to_os_environ
    raise subprocess.CalledProcessError(proc.returncode, _args)
subprocess.CalledProcessError: Command '['/bin/bash', '-o', 'errexit', '/anaconda/envs/compile/conda-bld/onnx_1636708558027/work/conda_build.sh']' returned non-zero exit status 1.
</code></pre>
<p>Any help is very appreciated,<br>
Greets from germany</p>

ptrblck: <p>The <a href="https://github.com/pytorch/builder">pytorch/builder</a> repository contains the build scripts for the pip wheels and conda binaries, so you could compare your setup to the official one and also use the docker container workflow for the build.</p>