Significant difference of performance between the extension that was compiled by g++ directly and that was built by "python3 setup.py install"

username: Max_xu
cooked: <p>Hi!</p>
<p>I wrote a cpp extension for torch which is a custom convolutional function.</p>
<p>Firstly, I compiled this function with g++ directly which was used for testing, the latency is <strong>5 milliseconds</strong>.</p>
<p>Secondly, I tried to integrate this function to torch and installed this extension by setuptools, following the steps shown in the tutorial provided by torch. However, the latency is now <strong>16 milliseconds</strong>.</p>
<p>The function invokation will consumes about 1-2 ms, so why the performance differs so much?</p>
<p>The compilation by g++ directly was done by</p>
<pre><code class="lang-auto">g++ -pthread -mavx2 -mfma ...
</code></pre>
<p>and the directives in the source file includes</p>
<pre><code class="lang-auto">#pragma GCC diagnostic ignored "-Wformat"

#pragma STDC FP_CONTRACT ON

#pragma GCC optimize("O3","unroll-loops","omit-frame-pointer","inline") //Optimization flags

// #pragma GCC option("arch=native","tune=native","no-zero-upper") //Enable AVX

#pragma GCC target("avx")
</code></pre>
<p>These directives were also included in the file built by setuptools. The “setup.py” file is</p>
<pre><code class="lang-auto">setup(
    name = 'cusconv_cpp',
    ext_modules=[
        CppExtension(name='cusconv_cpp', sources=['src/cusconv.cpp'],
        extra_compile_args={'cxx': ['-O3', '-pthread', '-mavx2', '-mfma']})
    ],
    cmdclass={
        'build_ext': BuildExtension
    })
</code></pre>
<p>The output log by setuptools for buiding is</p>
<pre><code class="lang-auto">x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/max/.local/lib/python3.6/site-packages/torch/lib/include -I/home/max/.local/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/max/.local/lib/python3.6/site-packages/torch/lib/include/TH -I/home/max/.local/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/include/python3.6m -c src/indconv.cpp -o build/temp.linux-x86_64-3.6/src/indconv.o -O3 -pthread -mavx2 -mfma -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=indconv_cpp -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
</code></pre>
<p>which indeed includes those flags but many other flags were also used.<br>
Anyone has any ideas?</p>

username: albanD
cooked: <p>Your timings are done at the same place in your c++ code in both cases? Or do you measure the time in python for one and in cpp for the other?</p>

username: Max_xu
cooked: <p>Thanks, I did timeing all in cpp, invoking same function, timing same area.</p>

username: Max_xu
cooked: <p>Another point, I tried to use “-O3” or “-O1” for extra_compile_flags in “setup.py”, there is no difference, always 16ms.<br>
Then I tried removing “-O3” in the source file <span class="hashtag">#pragmas</span> and using “g++” to compile it directly, the perfomance is now 35ms. Adding “-O1” will result in 6-7ms.</p>

username: Max_xu
cooked: <p>Removing all extra_compile_flags in “setup.py” will result in the same performance. It seems like those flags used  in “setup.py” doesn’t work, some default flags adopted by setuptools prevent further optimizations.</p>

username: Haydnspass
cooked: <p>Could you say a bit more regarding the flags?<br>
Usually the -O3 should be the highest level of optimisation, right?</p>

username: Max_xu
cooked: <p>After so many days, I still didn’t find any solutions to this. I tried all the same code in a old machine, no problem. Then now I tried again in another new machine, same problem happens.</p>
<p>my setup.py</p>
<pre><code class="lang-auto">import os
# os.environ['CC'] = 'g++'
# os.environ['CXX'] = 'g++'
from setuptools import setup
import sys
import re
from distutils import sysconfig
from distutils.core import setup, Extension

# if sys.platform == 'linux' or sys.platform == 'darwin':
#   sysconfig.get_config_var(None)  # to fill up _config_vars
#   d = sysconfig._config_vars
#   for x in ['OPT', 'CFLAGS', 'PY_CFLAGS', 'PY_CORE_CFLAGS', 'CONFIGURE_CFLAGS', 'LDSHARED']:
#     d[x] = re.sub(' -g ', ' ', d[x])
#     d[x] = re.sub('^-g ', '',  d[x])
#     d[x] = re.sub(' -g$', '',  d[x])
from torch.utils.cpp_extension import BuildExtension, CppExtension



setup(
    name = 'op_conv_ext',
    ext_modules=[
        CppExtension(name='op_conv_ext', sources=['extension/conv3x3s1.cpp'],
       extra_compile_args={'cxx': ['-O3', '-pthread', '-mavx2', '-mfma', '-funroll-loops', '-fomit-frame-pointer', '-finline-small-functions', '-finline-functions-called-once', 
           '-finline-functions', '-ffp-contract=on', "-mno-vzeroupper", '-m64', '-g', '-DNDEBUG[-march=native] [-mtune=native]' ]},
       extra_link_args=['-O3', '-pthread', '-mavx2', '-mfma', '-funroll-loops', 
           '-fomit-frame-pointer', '-finline-small-functions', '-finline-functions-called-once', 
           '-finline-functions', '-ffp-contract=on', "-mno-vzeroupper", '-m64', '-g', '-DNDEBUG[-march=native] [-mtune=native]' ],
    )],
    cmdclass={
        'build_ext': BuildExtension
    })

</code></pre>
<p>The output from <code>python3 setup.py</code> is</p>
<pre><code class="lang-auto">x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/zyx/.local/lib/python3.6/site-packages/torch/include -I/home/zyx/.local/lib/python3.6/site-packages/torch/include/torch/csrc/api/include -I/home/zyx/.local/lib/python3.6/site-packages/torch/include/TH -I/home/zyx/.local/lib/python3.6/site-packages/torch/include/THC -I/usr/include/python3.6m -c extension/conv3x3s1.cpp -o build/temp.linux-x86_64-3.6/extension/conv3x3s1.o -O3 -pthread -mavx2 -mfma -funroll-loops -fomit-frame-pointer -finline-small-functions -finline-functions-called-once -finline-functions -ffp-contract=on -mno-vzeroupper -m64 -g -DNDEBUG[-march=native] [-mtune=native] -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=op_conv_ext -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
x86_64-linux-gnu-g++ -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.6/extension/conv3x3s1.o -o build/lib.linux-x86_64-3.6/op_conv_ext.cpython-36m-x86_64-linux-gnu.so -O3 -pthread -mavx2 -mfma -funroll-loops -fomit-frame-pointer -finline-small-functions -finline-functions-called-once -finline-functions -ffp-contract=on -mno-vzeroupper -m64 -g -DNDEBUG[-march=native] [-mtune=native]
</code></pre>
<p>Flags in the source file is the same.</p>
<p>Now the performance invoked from python is <strong>35ms</strong>, that invoked from c++ which is built by g++ directly is <strong>25ms</strong>.</p>