Error when building custom CUDA kernels with PyTorch 1.6.0

username: jasrk11
cooked: <p>I’m unable to build the <a href="https://github.com/NVIDIA/flownet2-pytorch/" rel="noopener nofollow ugc">FlowNet 2.0</a> CUDA kernels for the layers <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/channelnorm_package" rel="noopener nofollow ugc">channelnorm</a>, <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/resample2d_package" rel="noopener nofollow ugc">resample2d</a>, <a href="https://github.com/NVIDIA/flownet2-pytorch/tree/master/networks/correlation_package" rel="noopener nofollow ugc">correlation</a> when using PyTorch &gt;= 1.5.1. However, I’m able to successfully build and use them with PyTorch &lt;= 1.4.0. Is there a way to make this work since I need to use PyTorch &gt;= 1.5.1?<br>
Following is a snippet of the long error log that I get:</p>
<blockquote>
<p>ninja: build stopped: subcommand failed.<br>
Traceback (most recent call last):<br>
File “/home/software/miniconda2/envs/cu101torch151s2/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 1423, in _run_ninja_build<br>
check=True)<br>
File “/home/software/miniconda2/envs/cu101torch151s2/lib/python3.6/subprocess.py”, line 438, in run<br>
output=stdout, stderr=stderr)<br>
subprocess.CalledProcessError: Command ‘[‘ninja’, ‘-v’]’ returned non-zero exit status 1.</p>
<p>During handling of the above exception, another exception occurred:</p>
<p>Traceback (most recent call last):<br>
File “setup.py”, line 32, in <br>
‘build_ext’: BuildExtension<br>
File “/home/software/miniconda2/envs/cu101torch151s2/lib/python3.6/site-packages/setuptools/<strong>init</strong>.py”, line 163, in setup<br>
return distutils.core.setup(**attrs)</p>
<p>.<br>
.<br>
.<br>
.</p>
<p>File “/home/software/miniconda2/envs/cu101torch151s2/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 1163, in _write_ninja_file_and_compile_objects<br>
error_prefix=‘Error compiling objects for extension’)<br>
File “/home/software/miniconda2/envs/cu101torch151s2/lib/python3.6/site-packages/torch/utils/cpp_extension.py”, line 1436, in _run_ninja_build<br>
raise RuntimeError(message)<br>
RuntimeError: Error compiling objects for extension</p>
</blockquote>
<p>For reproducing the error with PyTorch &gt;= 1.5.1 (installed using conda):</p>
<pre><code class="lang-auto"># get flownet2-pytorch source
git clone https://github.com/NVIDIA/flownet2-pytorch.git
cd flownet2-pytorch

# install custom layers
bash install.sh
</code></pre>

username: ptrblck
cooked: <p>Could you disable <code>ninja</code> for the build of the custom extension and post the stack trace with the error message here, please?</p>

username: jasrk11
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,  I disabled ninja for the build. The complete stack trace is too long for my own terminal but here are some of the error messages:</p>
<pre><code class="lang-auto">/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/modules/container/sequential.h: In member function ‘ReturnType torch::nn::SequentialImpl::forward(InputTypes&amp;&amp; ...)’:
/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/c10/util/Exception.h:333:9: error: ‘str’ is not a member of ‘c10’
         ::c10::str(__VA_ARGS__),                                      \
         ^
.....

/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/c10/util/TypeCast.h:57:58: error: ‘apply’ is not a member of ‘c10::maybe_real&lt;true, c10::complex&lt;double&gt; &gt;’
       static_cast&lt;int64_t&gt;(maybe_real&lt;real, src_t&gt;::apply(src)));
                                                          ^
.....

/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/c10/util/Optional.h:408:23: error: cannot bind ‘c10::intrusive_ptr&lt;torch::jit::InlinedCallStack&gt;’ lvalue to ‘c10::intrusive_ptr&lt;torch::jit::InlinedCallStack&gt;&amp;&amp;’
       contained_val() = std::forward&lt;U&gt;(v);
                       ^
/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/c10/util/Optional.h:408:23: error: no match for ‘operator=’ (operand types are ‘const std::shared_ptr&lt;torch::jit::Graph&gt;’ and ‘std::shared_ptr&lt;torch::jit::Graph&gt;’)

......

/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/torch/csrc/api/include/torch/nn/cloneable.h:69:56: error: no matching function for call to ‘at::Tensor::to(const c10::Device&amp;) const’
       auto data = device &amp;&amp; tensor.device() != *device ?
                                                        ^
</code></pre>

username: ptrblck
cooked: <p>Thanks for the stack trace.<br>
You could pipe the log output to a file in case your terminal gets flooded.<br>
That being said, if the posted error is the first one, I would assume that a stale build is creating this issue.<br>
Could you clean the build and update the submodules before trying to rebuild?</p>
<pre><code class="lang-auto">python setup.py clean
git submodule update --init --recursive
python setup.py install 2&gt;&amp;1 | tee install.log
</code></pre>

username: jasrk11
cooked: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks! I noticed that the first error in <code>install.log</code> was</p>
<p><code>/home/rakesh/software/miniconda2/envs/cu101torch16/lib/python3.6/site-packages/torch/include/c10/util/C++17.h:24:2: error: #error You need C++14 to compile PyTorch  #error You need C++14 to compile PyTorch</code></p>
<p>I changed the <code>cxx_args</code> in <code>setup.py</code> to <code>'-std=c++14'</code> which fixed the errors.</p>