CUDA_tensor_apply in extension gives undefined symbol

username: TomB
cooked: <p>I;m trying to use <code>CUDA_tensor_applyN</code> from <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/cuda/CUDAApplyUtils.cuh" rel="nofollow noopener"><code>ATen/cuda/CUDAApplyUtils.cuh</code></a> in an extension. While it builds fine when I try to import the extension I get:</p>
<pre><code class="lang-auto">ImportError: /root/local/miniconda/lib/python3.7/site-packages/mish_cuda/_C.cpython-37m-x86_64-linux-gnu.so:
undefined symbol: _ZN2at6native6legacy4cuda27_th_copy_ignoring_overlaps_ERNS_6TensorERKS3_
</code></pre>
<p>Demangled: <code>at::native::legacy::cuda::_th_copy_ignoring_overlaps_(at::Tensor&amp;, at::Tensor const&amp;)</code></p>
<p>I don’t think it’s a compiler compatability or linker options issue as similar posts tend to be, but I have very little c++ dev experience so could be wrong. I’ve built another extension that imports and runs fine on the same system (though different conda environment). I’ve tried with both conda installed toolchain (gxx_linux-64 7.3.0/binutils_linux-64 2.31.1) and system toolchain (Arch linux). I also tried in a docker based off of <a href="https://github.com/pytorch/extension-script" rel="nofollow noopener">pytorch/extension-script</a> though with <code>FROM nvidia/cuda:10.0-cudnn7-devel-ubuntu16.04</code> for CUDA support.</p>
<p>Poking around PyTorch libs in the docker I find:</p>
<pre><code class="lang-auto">$ objdump -t ${CONDA_PREFIX}/lib/python3.7/site-packages/torch/lib/libtorch.so\
  | grep '_ZN2at6native6legacy4cuda27_th_copy_ignoring_overlaps_ERNS_6TensorERKS3_'
0000000003bfed90 l     F .text  0000000000000552              _ZN2at6native6legacy4cuda27_th_copy_ignoring_overlaps_ERNS_6TensorERKS3_
$ objdump -T ${CONDA_PREFIX}/lib/python3.7/site-packages/torch/lib/libtorch.so \
  | grep '_ZN2at6native6legacy4cuda27_th_copy_ignoring_overlaps_ERNS_6TensorERKS3_'
$
</code></pre>
<p>I’m not using any custom compiler/linker options (except <code>--expt-extended-lambda</code> for nvcc). I did try explicitly linking against <code>libtorch.so</code> which didn’t help and I gather isn’t needed as <code>CUDAExtension</code> explicitly only links this on windows.</p>
<p>I nastily hacked out that function (only used for overlapping tensors) through a <code>#define</code> and the extension builds, imports and runs fine.</p>
<p>Thanks.</p>

username: tom
cooked: <p>This is terribly hard to debug remotely. For me, something like this seems to arise most often when the headers you use in C++ and the library (torch) Python uses are out of sync, i.e. have different versions.</p>
<p>Best regards</p>
<p>Thomas</p>

username: TomB
cooked: <p>Thanks a lot for replying. Completely understand the difficulty of debugging.</p>
<p>I’m building/linking against headers/libs out of the conda environment I’m testing in (and tried in a docker), so not quite sure how they’d be out of sync, unless I’m missing something there.</p>
<p>I have now put up the source for the extension. There’s a setup.py (and Dockerfile I also tested against). As noted the error is only on import so:</p>
<pre><code class="lang-auto">$ pip install git+https://github.com/thomasbrandon/mish-cuda &amp;&amp; python -c "import mish_cuda"
:Collecting git+https://github.com/thomasbrandon/mish-cuda
&lt;snip&gt;
ImportError: /home/user/dev/mish-cuda/.venv/lib/python3.7/site-packages/mish_cuda/_C.cpython-37m-x86_64-linux-gnu.so:
undefined symbol: _ZN2at6native6legacy4cuda27_th_copy_ignoring_overlaps_ERNS_6TensorERKS3_
</code></pre>
<p>And here’s an inline JIT reproduction (just directly using the function causing issues not the <code>CUDA_tensor_apply</code> which is actually what I care about): Gives me the same error as above.</p>
<pre><code class="lang-auto">#!/usr/bin/env python
from torch.utils.cpp_extension import load_inline
src = """
#include &lt;ATen/LegacyTHFunctionsCUDA.h&gt;
void import_test(torch::Tensor x, const torch::Tensor y) {
    auto z = at::native::legacy::cuda::_th_copy_ignoring_overlaps_(x, y);
}
"""
mod = load_inline(name='import_error_test', cpp_sources=src, functions='import_test')
</code></pre>
<p>I’ll also try and replicate against libtorch to see if it’s just an extension issue (though as it also looks to dynamically link I suspect not, statically linking may behave differently, on my minimal knowledge here). Just need to get on top of the process there.</p>
<p>One thing I’m a bit unclear on is exactly what is supposed to be externally usable in PyTorch, if I was more sure this was externally usable and it wasn’t a system config error at my end I’d submit an issue. Ideally I’d submit a PR but I struggle a bit to understand how exactly the declaration generation stuff works in PyTorch.<br>
It is defined in <a href="https://github.com/pytorch/pytorch/blob/3099732017239c3efce9b41f205a9c9df376fd5a/aten/src/ATen/Declarations.cwrap#L1887" rel="nofollow noopener">src/Aten/Declarations.cwrap</a> and does appear in the generated include/ATen/LegacyTHFunctionsCUDA.h.<br>
So just some confirmation there would be great. Or pointers about what parts in the generation process I might look at.</p>

username: tom
cooked: <p>Yes, so as far as I can see it is not part of the public API (none of the functions in that header are): it is not t declared <code>TORCH_API</code>.</p>
<p>Best regards</p>
<p>Thomas</p>

username: TomB
cooked: <p>Thanks a lot, hadn’t noticed that define. Does that apply to header only functions? It looks perhaps not as <code>PackedTensorAccessor</code> is not declared with that but I gather is public API. So then it’s a little unclear if <code>CUDA_tensor_apply</code> is intended to be private or not.</p>

username: tom
cooked: <p>Yeah, it doesn’t apply to header only functions. However, as you noticed, it relies on <code>_th_copy_ignoring_overlaps_</code> which isn’t exported.</p>
<p>So it could be that <code>CUDA_tensor_apply</code> is not that well-liked (with TensorIterator taking the part but not being public API), but I would not know.</p>
<p>If you are very keen to use it (and I can see how “have an utility for applying pointwise functions” is a very reasonable thing), maybe opening an issue asking about it is good.<br>
I could be that we just export the missing symbol and be done with it, or that there is another preferred solution.</p>
<p>Best regards</p>
<p>Thomas</p>

username: TomB
cooked: <p>Yeah, looking at <code>TORCH_API</code> I realised  it couldn’t be used with a header only function  as it controls import/export which is not appropriate there. I gather there’s no equivalent marking for header only functions. Though yes, using private functions would be indicative of not being intended to be public. I guess underlying this is me not really being sure about the include folder in the conda package (which seems to mirror the include folder in libtorch). For torch includes this looks to come from<br>
<a href="https://github.com/pytorch/pytorch/tree/master/torch/csrc/api/include/torch" rel="nofollow noopener"><code>torch/csrc/api/include/torch</code></a> but I’m not sure how the Aten and c10 stuff gets in here. My initial expectation everything in these includes was intended to be public seems wrong. I gather this might also be due to some things that were designed for internal use being now made public as libtorch and the C++ interface is developed.</p>
<p>Investigating the private function it looks like it’s use may be a legacy from when this was adapted from THC code. Following subsequent changes in ATen I think it can now be safely replaced with the standard <code>Tensor::copy_</code>. I think this is the only location this function is used in ATen so this would allow removal of the ATen wrapper which I gather would be desirable in the ongoing work to remove THC dependencies. The function is also a slightly nasty hack (as noted in code comments) and so nice to be able to avoid.<br>
Just need to confirm this a little more and I’ll submit an issue. This seems a stronger reason for removing it than allowing public use which while nice is likely not the primary concern of PyTorch devs. So hopefully a win/win.</p>
<p>Thanks again, and also thanks for your wonderful articles. The one on autograd functions and JIT operators in C++ in particular has been especially helpful in starting to get a handle on how they fit together which was previously eluding me.</p>