Unexpected results without clone (cpp extension)

username: zzd1992
cooked: <p>I write a cpp extension and it is successfully compiled and installed.<br>
I have a CUDA kernel like this:</p>
<pre><code class="lang-auto">__global__ void g(const float *x, float *y);
</code></pre>
<p>And a function with torch’s argument type like this:</p>
<pre><code class="lang-auto">torch::Tensor f(const torch::Tensor &amp;x) {
    auto y = torch::zeros_like(x);
    g&lt;&lt;&lt; 64, 1024, 0, at::cuda::getCurrentCUDAStream() &gt;&gt;&gt; 
        (x.clone().data&lt;float&gt;(), y.data&lt;float&gt;());
    return y;
}
</code></pre>
<p>This gives me expected results. However, when I remove <code>clone()</code>:</p>
<pre><code class="lang-auto">torch::Tensor f(const torch::Tensor &amp;x) {
    auto y = torch::zeros_like(x);
    g&lt;&lt;&lt; 64, 1024, 0, at::cuda::getCurrentCUDAStream() &gt;&gt;&gt;
        (x.data&lt;float&gt;(), y.data&lt;float&gt;());
    return y;
}
</code></pre>
<p>It gives me different and unexpercted results.<br>
I can’t understand why this happens because <code>x</code> is <code>const</code> in both <code>f</code> and <code>g</code>.<br>
Can someone tell me the reason of above phenomenon？</p>

username: tom
cooked: <p>You didn’t ask about this, using <code>x.clone().data&lt;float&gt;()</code> looks very fishy to me in terms of ownership of the memory pointer. RAII (i.e. memory lives as long as you have the object) means you need to assign <code>x.clone()</code> to a local variable (I thought that what you do was a compile error previously, but apparently I am mistaken). This is even more important given that CUDA is asynchronous.</p>
<p>What you likely do in your kernel (which you didn’t show, but which is likely where the error lies) is to expect contiguous tensors. The right thing to do is to define a local variable <code>auto xc = x.contiguous()</code><br>
If you need elementwise access, passing <code>PackedTensorAccessor</code>s to your kernels from <code>xc.packed_accessor&lt;...&gt;()</code> is a good way to pass tensors along with size and stride info.</p>
<p>Best regards</p>
<p>Thomas</p>

username: zzd1992
cooked: <p>Thanks so much.<br>
It works to set the variable <code>contiguous</code>.<br>
I am not familiar with <code>PackedTensorAccessors</code> and I can’t find related topics on the Internet. Can you provide an example of how to use it or some related materials?</p>

username: tom
cooked: <aside class="quote no-group" data-username="zzd1992" data-post="3" data-topic="59568">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zzd1992/48/7065_2.png" class="avatar"> zzd1992:</div>
<blockquote>
<p>I am not familiar with <code>PackedTensorAccessors</code> and I can’t find related topics on the Internet. Can you provide an example of how to use it or some related materials?</p>
</blockquote>
</aside>
<p>I should SEO optimize my blog more (it’s the second item on DuckDuckGo, but not on the first page at Google) <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> . <a href="https://lernapparat.de/sinkhorn-kernel/" rel="noopener nofollow ugc">My blog post on a Sinkhorn kernel uses and comments on them</a> with a full code example.<br>
I also used them for PyTorch’s own <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/cuda/Normalization.cuh" rel="noopener nofollow ugc">batch norm implementation</a>. That and the <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="noopener nofollow ugc">C++ extension tutoral</a> have the latest refactoring with <code>packed_accessor32</code> (32 bit indexing is <em>much</em> faster on CUDA than 64 bit).</p>
<p>Best regards</p>
<p>Thomas</p>