Illegal memory access on tensors with large dimensions for extension

username: 5had3z
cooked: <p>I am trying to use the correlation function from <a href="https://github.com/NVlabs/PWC-Net" rel="nofollow noopener">https://github.com/NVlabs/PWC-Net</a> and/or <a href="https://github.com/lliuz/ARFlow" rel="nofollow noopener">https://github.com/lliuz/ARFlow</a>. I have updated both to use the static method implementation required for 1.6.0 and replaced THCudaTensor/at::Tensor with torch::Tensor.</p>
<p>If I try to test with reasonably sized tensors (i.e. 16, 8, 128, 128), it fails with an illegal memory access during the backward methods (after resizing blobs), however it will work fine with smaller shaped tensors (i.e. 16, 64, 64, 64), even if the volume is greater.</p>
<p>I haven’t touched the CUDA kernels for either of them, and I guess they should be fine as they’re both published (and one being from nvidia)…</p>
<p>Both of them are previously used older versions of CUDA (8&amp;9) and Torch (0.4 and 1.1), could this also be the issue?</p>
<p>Source for my changes can be found here.<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/5had3z/stereo-to-all/tree/debug/correlation_pkg/nnet_training_framework/correlation_package" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/59598353?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/5had3z/stereo-to-all/tree/debug/correlation_pkg/nnet_training_framework/correlation_package" target="_blank" rel="nofollow noopener">5had3z/stereo-to-all</a></h3>

<p>The aim of this project is to generate a pointcloud with motion vectors and semantic segmentation of stereoscopic image sequences. Targeting real-time deployment on an nVIDIA Xavier. - 5had3z/stere...</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>Cheers</p>

username: ptrblck
cooked: <p>Your link is not available and yields a 404.<br>
I don’t know which custom kernels you are using, but since the illegal memory access seems to be size-dependent, I guess some int32/int64 indexing might fail.<br>
You could run the code via <code>cuda-gdb</code> and create an issue in the corresponding repository, where the kernel is provided.</p>

username: 5had3z
cooked: <p>I did a few more tests and debugging via printing in the kernels, and it seems for some combinations of input height/width and padding (moreso lack of padding), during the correlation forward and backward, it tries to access outside of the image i.e. index -1.</p>
<p>I’m not entirely sure who is the original author of the correlation kernel, I think the earliest I’ve noticed is nvidia’s PWC-net. Its 404’ing because I figured this out, made the fix and noted that I need at least some padding (and to increase it if I have this problem again) and then merged it back into my master branch.</p>
<p>The bad accesses occur where the boundary checks are (unless otherwise stated in a comment). If I can figure out a more elegant solution, I could make one and submit a PR? I believe there should probably be a way of calculating required minimal required padding.<br>
</p><aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/5had3z/stereo-to-all/blob/master/nnet_training/correlation_package/correlation_cuda_kernel.cu" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/5had3z/stereo-to-all/blob/master/nnet_training/correlation_package/correlation_cuda_kernel.cu" target="_blank" rel="nofollow noopener">5had3z/stereo-to-all/blob/master/nnet_training/correlation_package/correlation_cuda_kernel.cu</a></h4>
<pre><code class="lang-cu">#include "correlation_cuda_kernel.cuh"

#define CUDA_NUM_THREADS 1024
#define THREADS_PER_BLOCK 32

#include &lt;ATen/Dispatch.h&gt;
#include &lt;ATen/cuda/CUDAContext.h&gt;
#include &lt;cuda_runtime.h&gt;

// using at::Half;
#define TensorAcc4R torch::PackedTensorAccessor32&lt;scalar_t,4,torch::RestrictPtrTraits&gt;

template &lt;typename scalar_t&gt;
__global__ void channels_first(const TensorAcc4R input, TensorAcc4R rinput,
	const int channels, const int height, const int width, const int pad_size)
{
	// n (batch size), c (num of channels), y (height), x (width)
	const int n = blockIdx.x;
	const int y = blockIdx.y;
	const int x = blockIdx.z;
</code></pre>

  This file has been truncated. <a href="https://github.com/5had3z/stereo-to-all/blob/master/nnet_training/correlation_package/correlation_cuda_kernel.cu" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: ptrblck
cooked: <p>That sounds like great debugging. Sure, if you have a proper fix, the authors would probably be really glad to receive it. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>