Is building from source really speed up inference speed?

username: be7f984b2f2e66ba7969
cooked: <p>Hi,</p>
<p>I tried to build from source of pytorch on gtx-1070ti.</p>
<p>python setup.py build develop</p>
<p>but found no inference speed up, the results seem to be opposite to this post (<a href="https://medium.com/repro-repo/build-pytorch-from-source-on-ubuntu-18-04-1c5556ca8fbf" rel="nofollow noopener">https://medium.com/repro-repo/build-pytorch-from-source-on-ubuntu-18-04-1c5556ca8fbf</a>)</p>
<p>any idea?</p>

username: albanD
cooked: <p>Hi,</p>
<p>The binary comes with a set of bundled libraries (mkl, magma, etc) that are very important for speed.<br>
If you compile from source, you will want to make sure you have the relevant ones for your workload installed locally so that they can be used during compilation.<br>
Also in some cases, build of openBLAS, magma, etc tailored to your machine will be slightly faster.<br>
But if you don’t have any blas library (or use the system default one) you might see lower performance compared to the binary code that has an off the shelf optimized blas for example.</p>

username: be7f984b2f2e66ba7969
cooked: <p>pytorch has many third party libraries,  but i  am not sure which one is the key library to speed up the performance.</p>
<p>Do you have any suggestion that how to enable these libraries when compiling?</p>

username: be7f984b2f2e66ba7969
cooked: <p>Here are my cmake output, which library I have to enable in order to speed up the inference?</p>
<h2>– ******** Summary ********<br>
– General:<br>
–   CMake version         : 3.15.4<br>
–   CMake command         : /home/acer/cmake-3.15.4-Linux-x86_64/bin/cmake<br>
–   System                : Linux<br>
–   C++ compiler          : /usr/bin/c++<br>
–   C++ compiler id       : GNU<br>
–   C++ compiler version  : 7.5.0<br>
–   BLAS                  : MKL<br>
–   CXX flags             :  -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow<br>
–   Build type            : Release<br>
–   Compile definitions   : ONNX_ML=1;ONNXIFI_ENABLE_EXT=1;ONNX_NAMESPACE=onnx_torch;HAVE_MMAP=1;_FILE_OFFSET_BITS=64;HAVE_SHM_OPEN=1;HAVE_SHM_UNLINK=1;HAVE_MALLOC_USABLE_SIZE=1;USE_EXTERNAL_MZCRC;MINIZ_DISABLE_ZIP_READER_CRC32_CHECKS<br>
–   CMAKE_PREFIX_PATH     : /home/acer/.pyenv/versions/pytorch_build/lib/python3.7/site-packages;/usr/local/cuda<br>
–   CMAKE_INSTALL_PREFIX  : /home/acer/nfs-share/pytorch/torch</h2>
<p>–   TORCH_VERSION         : 1.7.0<br>
–   CAFFE2_VERSION        : 1.7.0<br>
–   BUILD_CAFFE2_MOBILE   : OFF<br>
–   USE_STATIC_DISPATCH   : OFF<br>
–   BUILD_BINARY          : OFF<br>
–   BUILD_CUSTOM_PROTOBUF : ON<br>
–     Link local protobuf : ON<br>
–   BUILD_DOCS            : OFF<br>
–   BUILD_PYTHON          : True<br>
–     Python version      : 3.7.4<br>
–     Python executable   : /home/acer/.pyenv/versions/pytorch_build/bin/python<br>
–     Pythonlibs version  : 3.7.4<br>
–     Python library      : /home/acer/.pyenv/versions/3.7.4/lib/libpython3.7m.so.1.0<br>
–     Python includes     : /home/acer/.pyenv/versions/3.7.4/include/python3.7m<br>
–     Python site-packages: lib/python3.7/site-packages<br>
–   BUILD_CAFFE2_OPS      : ON<br>
–   BUILD_SHARED_LIBS     : ON<br>
–   BUILD_TEST            : True<br>
–   BUILD_JNI             : OFF<br>
–   INTERN_BUILD_MOBILE   :<br>
–   CLANG_CODE_COVERAGE   : OFF<br>
–   USE_ASAN              : OFF<br>
–   USE_CUDA              : ON<br>
–     CUDA static link    : OFF<br>
–     USE_CUDNN           : ON<br>
–     CUDA version        : 10.2<br>
–     cuDNN version       : 7.6.5<br>
–     CUDA root directory : /usr/local/cuda<br>
–     CUDA library        : /usr/local/cuda/lib64/stubs/libcuda.so<br>
–     cudart library      : /usr/local/cuda/lib64/libcudart.so<br>
–     cublas library      : /usr/lib/x86_64-linux-gnu/libcublas.so<br>
–     cufft library       : /usr/local/cuda/lib64/libcufft.so<br>
–     curand library      : /usr/local/cuda/lib64/libcurand.so<br>
–     cuDNN library       : /usr/lib/x86_64-linux-gnu/libcudnn.so<br>
–     nvrtc               : /usr/local/cuda/lib64/libnvrtc.so<br>
–     CUDA include path   : /usr/local/cuda/include<br>
–     NVCC executable     : /usr/local/cuda/bin/nvcc<br>
–     NVCC flags          : -DONNX_NAMESPACE=onnx_torch;-gencode;arch=compute_61,code=sm_61;-Xcudafe;–diag_suppress=cc_clobber_ignored;-Xcudafe;–diag_suppress=integer_sign_change;-Xcudafe;–diag_suppress=useless_using_declaration;-Xcudafe;–diag_suppress=set_but_not_used;-Xcudafe;–diag_suppress=field_without_dll_interface;-Xcudafe;–diag_suppress=base_class_has_different_dll_interface;-Xcudafe;–diag_suppress=dll_interface_conflict_none_assumed;-Xcudafe;–diag_suppress=dll_interface_conflict_dllexport_assumed;-Xcudafe;–diag_suppress=implicit_return_from_non_void_function;-Xcudafe;–diag_suppress=unsigned_compare_with_zero;-Xcudafe;–diag_suppress=declared_but_not_referenced;-Xcudafe;–diag_suppress=bad_friend_decl;-std=c++14;-Xcompiler;-fPIC;–expt-relaxed-constexpr;–expt-extended-lambda;-Wno-deprecated-gpu-targets;–expt-extended-lambda;-gencode;arch=compute_61,code=sm_61;-Xcompiler;-fPIC;-DCUDA_HAS_FP16=1;-D__CUDA_NO_HALF_OPERATORS__;-D__CUDA_NO_HALF_CONVERSIONS__;-D__CUDA_NO_HALF2_OPERATORS__<br>
–     CUDA host compiler  : /usr/bin/cc<br>
–     NVCC --device-c     : OFF<br>
–     USE_TENSORRT        : OFF<br>
–   USE_ROCM              : OFF<br>
–   USE_EIGEN_FOR_BLAS    : ON<br>
–   USE_FBGEMM            : ON<br>
–     USE_FAKELOWP          : OFF<br>
–   USE_FFMPEG            : OFF<br>
–   USE_GFLAGS            : OFF<br>
–   USE_GLOG              : OFF<br>
–   USE_LEVELDB           : OFF<br>
–   USE_LITE_PROTO        : OFF<br>
–   USE_LMDB              : OFF<br>
–   USE_METAL             : OFF<br>
–   USE_MKL               : OFF<br>
–   USE_MKLDNN            : ON<br>
–   USE_MKLDNN_CBLAS      : OFF<br>
–   USE_NCCL              : ON<br>
–     USE_SYSTEM_NCCL     : OFF<br>
–   USE_NNPACK            : ON<br>
–   USE_NUMPY             : ON<br>
–   USE_OBSERVERS         : ON<br>
–   USE_OPENCL            : OFF<br>
–   USE_OPENCV            : OFF<br>
–   USE_OPENMP            : ON<br>
–   USE_TBB               : OFF<br>
–   USE_VULKAN            : OFF<br>
–   USE_PROF              : OFF<br>
–   USE_QNNPACK           : ON<br>
–   USE_PYTORCH_QNNPACK   : ON<br>
–   USE_REDIS             : OFF<br>
–   USE_ROCKSDB           : OFF<br>
–   USE_ZMQ               : OFF<br>
–   USE_DISTRIBUTED       : ON<br>
–     USE_MPI             : ON<br>
–     USE_GLOO            : ON<br>
–     USE_TENSORPIPE      : ON</p>

username: albanD
cooked: <aside class="quote no-group" data-username="be7f984b2f2e66ba7969" data-post="3" data-topic="91057">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/be7f984b2f2e66ba7969/48/6383_2.png" class="avatar"> be7f984b2f2e66ba7969:</div>
<blockquote>
<p>i am not sure which one is the key library to speed up the performance.</p>
</blockquote>
</aside>
<p>It all depend what is your workload <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"><br>
Do you use cpu or gpu? Do you use distributed? Do you use mixed precision? Do you use linear algebra functions?</p>

username: be7f984b2f2e66ba7969
cooked: <p>I use nvidia gpu and distributed, also mixed precision. and of course linear algebra functions.</p>

username: albanD
cooked: <p>In that case you want to make sure you have the latest cuda/cudnn installed.<br>
For distributed, the gloo backend should be already properly provided.<br>
For mixed precision I think basic cuda libs handle that well.<br>
For linear algebra. You want a good blas/lapack lib on cpu like mkl or openblas. And for gpu you will need magma.</p>
<p>You can find some instructions here: <a href="https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md">https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md</a></p>